Abstractions for Programming

Introduction
 Most of the current commercially available multimedia
applications are implemented in procedure-oriented programming
languages (structural languages) such as C.
 In the past, multimedia-specific functions were called, and
respectively controlled, through hardware-specific libraries or
device drivers.
 The programmers mostly use the so-called built-in functions of
higher programming languages to make use of floating point
processing capabilities.
 Also, interfaces to communication systems are often
implemented via object-oriented approaches
 Multimedia can become an integral part of the programming
environment if the proper class hierarchy is used. There is no
generally accepted optimal class hierarchy.
Abstraction Levels
 Abstraction levels in programming define different
approaches with a varying degree of detail for representing,
accessing and manipulating data.
 A device for processing continuous media can exist as a
separate component in a computer. In this case, a device
is not part of the operating system, but is directly
accessible to every component and application.
 A library, the simplest abstraction level, includes the
necessary functions for controlling the corresponding
hardware with specific device access operations.
 As with any device, multimedia devices can be bound
through a device driver, respectively the operating system.
Hence, the processing of the continuous data becomes part
of the system software.
 Dedicated programming languages, such as programming for
Digital Signal Processing (DSP), allow for the implementation of
real-time programs.
 The corresponding program mostly runs in a Real-Time
Environment (RTE) separate from the actual application. It is not
very common today that an application software is
programmed in the RTE.
 Higher procedural programming languages build the next
abstraction level. They are the languages most often used to
implement commercial multimedia applications.
 More flexibility for the programmer is provided via the
abstraction level - an object oriented environment. This
environment provides the application with a class hierarchy for
the manipulation of multimedia.
Libraries
 The processing of continuous media is based on a set of functions
which are embedded into libraries. This is the usual solution for
programming multimedia data. These libraries are provided
together with the corresponding hardware.
 The device driver and/or library, which controls all available
functions, also supports each device.
 The libraries differ very much in their degree of abstraction.
Some libraries can be considered as extensions of the graphical
user interface, whereas other libraries consist of control
instructions passed as control blocks to the corresponding
driver.
 Libraries are very useful at the operating system level, but there
is no agreement (and may never be) over which functions are
best for different drivers, i.e., which functions should be
supported.
System Software
 Instead of implementing access to multimedia devices
through individual libraries, the device access can become
part of the operating system.
 An example of access to multimedia devices and support
for continuous media processing implemented in operating
system is the experimental Nemo system from the
University of Cambridge
 The Nemo system consists of the Nemo Trusted
Supervisor Call, running in supervisor mode, and three
domains running in user mode: system, device driver and
application.
 The Nemo Trusted Supervisor Call (NTSC) code implements
those functions which are required by user mode
processes. It provides support for three types of
processes.
 System processes implement the majority of the services
provided by the operating system.
 Device driver processes are similar to system processes but
are distinguished by the fact that they are attached to device
interrupt stubs which execute in supervisor mode.
 Application processes contain user programs.
System Software Toolkits
 A simpler approach (from the user perspective) in a
programming environment than the system software
interface for control of the audio and video data
processing can be taken by using toolkits.
 These toolkits are used to:
 Abstract from the actual physical layer (it is also done in a
limited way by the libraries).
 Allow a uniform interface for communication with all different
devices of continuous media (with eventual input of quality of
service parameters).
 Introduce the client-server paradigm (here, the communication
can be hidden from the application in an elegant way).
 Toolkits can also hide process-structures. It would be of
great value for the development of multimedia application
software to have the same toolkit on different system
platforms, but according to current experiences, this
remains to be a wish, and it would cause a decrease in
performance.
 Toolkits should represent interfaces at the system
software level. In this case, it is possible to embed them
into the programming languages or object-oriented
environment.
Higher Programming Languages
 Procedural higher programming languages will be called High-
Level Languages (HLL).
 In such an HLL, the processing of continuous media data is
influenced by a group of similar constructed functions. These calls
are mostly hardware and driver-independent.
 Hence, their integration in HLLs leads to a wishful abstraction,
supports a better programming style and increases the
productivity.
 Programs must be capable of supporting and effectively
manipulating multimedia data.
 Therefore, the programs in an HLL either directly access
multimedia data structures, or communicate directly with the
active processes in a real-time environment.
 The processing devices are controlled through corresponding
device drivers.
 Compiler, linker and/or loader provide the required
communication between the application program and the
processing of continuous data.
 Media can be considered differently inside a
programming language.
Media as Types
 Example: OCCAM-2 was derived from Communication
Sequential Processes (CSP)
Media as Files
 Another possibility of programming continuous media
data is the consideration of continuous media streams as
files instead of data types.
Media as Processes
 The processing of continuous data contains a time-
dependency because the life span of a process equals to
the life span of a connection(s) between source(s) and
destination(s). A connection can exist locally, as well as
remotely.
 Under this consideration, it is possible to map
continuous media to processes nd to integrate them in
an HLL.
Media as Files

 In the above example, the process cont-process-a implements a

set of actions (functions) which apply to a continuous data
stream, Two of them are the modification of the volume set-
volume and the process of setting a volume, dependent from a
band filter, set-loudness.
 the processing can be done either once or continuously,
meaning that during the entire transmission of continuous
data:
 The loudness is determined once by a device driver call. The
driver loads a certain storage content which is used by the
running process controlling an audio board.
 If the main processor passes the audio data further from a file
to the communication system, then the loudness can be
changed here. Thus, the compression and coding must be
considered.
Programming Language Requirements
 The processing of continuous data is:
 Controlled by the HLL through pure asynchronous
instructions (typically, through the use of a library).
 An integral part of a program through the identification of the
media, respectively data streams with data types, variables, files
or processes.
 Therefore, the HLL should support a parallel processing
as was presented in all examples of HLL programming of
continuous media. Thus, it is of secondary importance if
the number of processes is known at compile time or if it
is defined dynamically at run-time.
 Interprocess Communication Mechanism
 Different processes must be able to communicate through an
Inter-Process Communication mechanism (IPC). This IPC
mechanism must be able to transmit audio and video in a
timely fashion because these media have a limited life span.
Therefore, the IPC must be able to:
 Understand a priori and/or implicitly specified time
requirements. These requirements can be specified using QoS
parameters or they can be extracted from the data type (if a
medium is implemented as a data type).
 Transmit the continuous data according to the requirements.
 Initiate the processing of the received continuous process on
time.
 Audio and video processes require the availability of real-time
processing.
 Language
 The authors see no demand for the development of a new
dedicated language. A partial language replacement is also quite
difficult because cooperation between the real-time
environment and the remaining programs requires semantic
changes in the programming languages.
 The IPC must be designed and implemented in real-time.
 A language extension is the solution proposed here. For the
purpose of simplicity, in the first step, a simple language should
be developed which satisfies most of the above described
requirements.
 An example of such a language is OCCAM-2. Some real-time
systems are implemented in this parallel programming
language.
Object-oriented Approaches
 The object-oriented approach was first introduced as a
method for the reduction of complexity in the software
development and it is used mainly with this goal today.
 Further, the reuse of software components is a main
advantage of this paradigm.
 The basic ideas of object-oriented programming are: data
encapsulation and inheritance, in connection with class and.
object definitions.
 The programs are implemented, instead of using functions
and data structures, by using classes, objects, and methods.
Abstract Type Definition
 The definition of data types through abstract interfaces is called
abstract type definitions.
 The abstract type definition is understood as an interface
specification without a knowledge and implementation of internal
algorithms.
 This data abstraction hides the used algorithm.
 In a distributed multimedia system, abstract data types are
assumed for virtual and real device units such as cameras and
monitors.
 For example, an interface, which contains a function zoom, can
also contain a parameter which specifies the actual position in
an area from 10 ... 500. However, this specification does not
describe the actual implementation.
Class
 The implementation of abstract data types is done
through classes.
 A class specification includes an interface provided to the
outside world.
 For example, in a class professional-camera, the
operations zoom and set-back-light are defined and
implemented. If the objects, which represent a closed
class, use only relative position entries, the
implementation of the zoom operation needs to
transform the absolute values into the necessary relative
parameters.
Object
 An object is the instance of the class. Therefore, all objects,
derived from the same class include the same operations as an
interface to the outside world.
 An object is created at run-time of the system. It includes a set
of operations, which are called methods.
 Objects communicate among each other through the exchange
of messages. Thus, a message calls the corresponding method of
the target object.
 In a distributed multimedia environment, virtual units are
considered to be objects.
 Thus, corresponding methods represent operations on the
devices. The method play of a VCR object (Video Cassette
Recorder) is mapped to the play-operation of the
corresponding VCR device driver.
 Multimedia data units (images, audio and video clips) can also
be considered objects.
Inheritance
 One of the most important properties of object-oriented
systems is inheritance.
 Classes contain, besides the root and leaves of the hierarchy,
super classes and subclasses (fathers and sons).
 For example, let the class professional-camera be a subclass of
the class camera. Methods such as autofocus-on and focus are
defined in the class camera. The professional-camera class also
has the method zoom. An object, which is derived from the
professional-camera, can use the method zoom, as well as the
operations focus and, autofocus-on.
 The main problem has been and remains to be the design of a
clear and uniform class hierarchy for a multimedia system.
Polymorphism
 Polymorphism is related to the property of inheritance
indicating when the same name of a method is defined in
several classes (and objects) with different implementations
and functionalities.
 For example, the function play is used with audio and
video data. It uses different device units for each medium.
The data can come either from a file of a local file
environment or from an audio-video sequence of an
external device.
 Inside of the object-oriented approach, for example, play
is defined in different classes. According to which object
must perform the operation, the corresponding method
is chosen.
 On the other hand, polymorphism can also very easily
cause programming errors that are difficult to find. Hence,
this abstraction strongly complicates the implementation.
 In an object-oriented environment, according to
Wegener's definition, multimedia programming is achieved
through the implementation and extension of class
hierarchies.
Application-specific Metaphors as Classes
 An application-specific class hierarchy introduces abstractions
specifically designed for a particular application. Thus, it is
not necessary to consider other class hierarchies.
 This approach leads to a number of different class
hierarchies. Furthermore, using this approach, one very
easily abandons the actual advantage of object-oriented
programming, i.e., the reuse of existing code.
 Although, for similar applications, similar class hierarchies
can be implemented. Therefore, a catalog of similar
applications is necessary to use the existing knowledge
for the development of a new application.
Application-generic Metaphors as Classes
 Another approach is to combine similar functionalities of all
applications. These properties or functions, which occur
repeatedly, can be defined and implemented as classes for all
applications.
 An application is defined only through a binding of this class.
For example, basic functions or functional units can create
classes.
 The methods of these classes inherit the general methods
through integration of application-specific subclasses.
 In theory this approach sounds easy to follow. In practice, we
have not yet a very useful set of basic/generic application
classes looks like, because known implementations of
application-generic classes only work well for a very restricted
set of applications
Devices as Classes
 Consider objects which reflect a physical view of the
multimedia system. The devices are assigned to objects which
represent their behavior and interface.
 Methods with similar semantics, which interact with different
devices, should be defined in a device-independent manner.
 The considered methods use internally, for example, methods
like start, stop and seek. Some units can manipulate several
media together.
 In a multimedia system, abstract device definitions can be
provided, e.g., camera and monitor. We did not say anything
until now about the actual implementation.
 The results show that defining a general and valid interface for
several similar audio and video units, as well as input and output
units, is quite a difficult design process.
 The concept of devices as class hierarchies provides a simple
parallel performance of the methods. Note, synchronization is
not supported in this hierarchy and must be provided through
other components; multiple inheritance is often needed
Processing Units as Classes
 This abstraction comprises source objects, destination
objects and combined source destination objects which
perform intermediate processing of continuous data.
 With this approach, a kind of "lego" system is created
which allows for the creation of a data flow path through a
connection of objects. The outputs of objects are connected
with inputs of other objects, either directly or through
channel objects.
 As an example of this concept, the processing unit as a
class is presented. It was used at the beginning of the
application implementation of a remote camera control
system.
Multimedia Object
 A multimedia application processes and controls (respectively
generates) the interactions and information of different continuous and
discrete media.
 From the object-oriented viewpoint, an application is considered to
be a multimedia object.
 Such an object uses or consists of many other objects which
contribute to the solution of the task.
 These objects are connected to, for example, representation of
different media and device units.
 Such a Compound Multimedia Object (CMO) consists of other
CMOs and, Basic Multimedia Objects (BMOs).
 The Basic Multimedia Class (BMC) typically represents an individual
medium of an input type (e.g., data from a camera, stored audio
sequences from a file) or output type (e.g., data output to a video
window or speaker)
 A Compound Multimedia Class (CMC) can control, and respectively
represent several media and devices.
 BMOs are instances of BMCs; CMOs are instances of CMCs.
Media as Classes
 The Media Class
Hierarchy defines a
hierarchical relation
for different media.
 The following
example shows such
a class hierarchy.
 Some programming environments already provide
languages based on the object oriented approach with
processing units or media as classes.
 For example: Gibbs' Multimedia Programming Environment
has a strong media relation. It is implemented through a
scripting language extension at the user interface.
Communication-specific Metaphors as
Classes
 Communication-oriented approaches often consider objects
in a distributed environment through an explicit
specification of classes and objects tied to a
communication system.
 Blakowski specifies, for example, information, presentation
and transport classes.
 The information, contained in the information objects, can
build a presentation object which is later used for
presentation of information.
 Information objects can be converted to transport
objects for transmission purposes.
 For further reading please visit:
http://ce.sharif.ir/courses/84-
85/2/ce342/resources/root/BOOK/Multimedia/215814-
%20Chapter%2016.pdf