Biological neural systems must grow their own connections and maintain topological relations betw... more Biological neural systems must grow their own connections and maintain topological relations between elements that are related to the sensory input surface. Artificial systems have traditionally prewired such maps, but the sensor arrangement is not always known and can be expensive to specify before run time. Here we present a method for learning and updating topographic maps in systems comprising modular, event-based elements. Using an unsupervised neural spike-timing-based learning rule combined with Hebbian learning, our algorithm uses the spatiotemporal coherence of the external world to train its network. It improves on existing algorithms by not assuming a known topography of the target map and includes a novel method for automatically detecting edge elements. We show how, for stimuli that are small relative to the sensor resolution, the temporal learning window parameters can be determined without using any user-specified constants. For stimuli that are larger relative to the sensor resolution, we provide a parameter extraction method that generally outperforms the small-stimulus method but requires one user-specified constant. The algorithm was tested on real data from a 64 × 64-pixel section of an event-based temporal contrast silicon retina and a 360-tile tactile luminous floor. It learned 95.8% of the correct neighborhood relations for the silicon retina within about 400 seconds of real-world input from a driving scene and 98.1% correct for the sensory floor after about 160 minutes of human pedestrian traffic. Residual errors occurred in regions receiving little or ambiguous input, and the learned topological representations were able to update automatically in response to simulated damage. Our algorithm has applications in the design of modular autonomous systems in which the interfaces between components are learned during operation rather than at design time.
Human pose estimation has dramatically improved thanks to the continuous developments in deep lea... more Human pose estimation has dramatically improved thanks to the continuous developments in deep learning. However, marker-free human pose estimation based on standard frame-based cameras is still slow and power hun-gry for real-time feedback interaction because of the huge number of operations necessary for large Convolutional Neural Network (CNN) inference. Event-based cameras such as the Dynamic Vision Sensor (DVS) quickly output sparse moving-edge information. Their sparse and rapid output is ideal for driving low-latency CNNs, thus poten-tially allowing real-time interaction for human pose estima-tors. Although the application of CNNs to standard frame-based cameras for human pose estimation is well estab-lished, their application to event-based cameras is still un-der study. This paper proposes a novel benchmark dataset of human body movements, the Dynamic Vision Sensor Hu-man Pose dataset (DHP19). It consists of recordings from 4 synchronized 346x260 pixel DVS cameras, for a set of 33 movements with 17 subjects. DHP19 also includes a 3D pose estimation model that achieves an average 3D pose estimation error of about 8 cm, despite the sparse and re-duced input data from the DVS.
We conceived Ada: Intelligent Space exhibit as an artificial organism, integrating a large number... more We conceived Ada: Intelligent Space exhibit as an artificial organism, integrating a large number of sensory modalities, and let it interact with visitors using a multitude of effector systems. Ada used a language of sound and light to communicate its moods, emotions, and behaviors. Here we describe the mechanisms behind Ada's sound communication, its real-time performance, and its interpretation by human subjects.
This paper and the accompanying video describe Ada, an immersive 160m 2 space for human-machine i... more This paper and the accompanying video describe Ada, an immersive 160m 2 space for human-machine interaction. Ada is conceived as an artificial organism that plays with its visitors, with multi-modal sensors consisting of pressure-sensitive floor tiles, microphones for localising and recognising sounds, and pan-tilt video cameras called gazers. Her output modalities include a 360º video projection screen capable of showing realtime computer graphics combined with multiple live video streams, RGB neon lights in the floor tiles, pan-tilt lights and an automatic neural music composition system. Visitors provided input to the space primarily by moving around, clapping their hands and speaking. Ada was presented at the Swiss national exhibition Expo.02 in Neuchâtel from May 15 to October 20, 2002, and attracted 553,700 visitors. In this paper we give a brief overview of the Ada exhibit, and describe the accompanying video.
Fast neuromorphic event-based vision sensors (Dynamic Vision Sensor, DVS) can be combined with sl... more Fast neuromorphic event-based vision sensors (Dynamic Vision Sensor, DVS) can be combined with slower conventional frame-based sensors to enable higher-quality inter-frame interpolation than traditional methods relying on fixed motion approximations using e.g. optical flow. In this work we present a new, advanced event simulator that can produce realistic scenes recorded by a camera rig with an arbitrary number of sensors located at fixed offsets. It includes a new configurable frame-based image sensor model with realistic image quality reduction effects, and an extended DVS model with more accurate characteristics. We use our simulator to train a novel reconstruction model designed for end-to-end reconstruction of high-fps video. Unlike previously published methods, our method does not require the frame and DVS cameras to have the same optics, positions, or camera resolutions. It is also not limited to objects a fixed distance from the sensor. We show that data generated by our simulator can be used to train our new model, leading to reconstructed images on public datasets of equivalent or better quality than the state of the art. We also show our sensor generalizing to data recorded by real sensors.
This paper describes a mathematical formulation for measuring how one system can estimate the con... more This paper describes a mathematical formulation for measuring how one system can estimate the consciousness of another. This consciousness estimate is always relative to the observer. The paper shows how this formulation leads to simple resolutions of some key problems of consciousness.
Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems, 2018
EVA 1 is describing a new class of emotion-aware autonomous systems delivering intelligent person... more EVA 1 is describing a new class of emotion-aware autonomous systems delivering intelligent personal assistant functionalities. EVA requires a multidisciplinary approach, combining a number of critical building blocks into a cybernetics systems/software architecture: emotion aware systems and algorithms, multimodal interaction design, cognitive modelling, decision making and recommender systems, emotion sensing as feedback for learning, and distributed (edge) computing delivering cognitive services.
Proceedings Fourth Annual Conference on Mechatronics and Machine Vision in Practice
A unique distributed cont developed to allow walk. It models competing directi robot to walk as q... more A unique distributed cont developed to allow walk. It models competing directi robot to walk as quick over given minimal in architecture features arrangement of cooper controller. This h conventional control scope for incremental facilitate accurate Accelerated learning robot's intrinsic knowledg infer additional info This was found to improve the rob particularly at the s information was limited. These concepts can be applied to autonomous robots for use in deep sea and other inhospitable environments.
Background: In recent years, virtual reality has been introduced to neurorehabilitation, in parti... more Background: In recent years, virtual reality has been introduced to neurorehabilitation, in particular with the intention of improving upper-limb training options and facilitating motor function recovery. Methods/Design: The proposed study incorporates a quantitative part and a qualitative part, termed a mixed-methods approach: (1) a quantitative investigation of the efficacy of virtual reality training compared to conventional therapy in upper-limb motor function are investigated, (2a) a qualitative investigation of patients' experiences and expectations of virtual reality training and (2b) a qualitative investigation of therapists' experiences using the virtual reality training system in the therapy setting. At three participating clinics, 60 patients at least 6 months after stroke onset will be randomly allocated to an experimental virtual reality group (EG) or to a control group that will receive conventional physiotherapy or occupational therapy (16 sessions, 45 minutes each, over the course of 4 weeks). Using custom data gloves, patients' finger and arm movements will be displayed in real time on a monitor, and they will move and manipulate objects in various virtual environments. A blinded assessor will test patients' motor and cognitive performance twice before, once during, and twice after the 4-week intervention. The primary outcome measure is the Box and Block Test. Secondary outcome measures are the Chedoke-McMaster Stroke Assessments (hand, arm and shoulder pain subscales), the Chedoke-McMaster Arm and Hand Activity Inventory, the Line Bisection Test, the Stroke Impact Scale, the MiniMentalState Examination and the Extended Barthel Index. Semistructured face-to-face interviews will be conducted with patients in the EG after intervention finalization with a focus on the patients' expectations and experiences regarding the virtual reality training. Therapists' perspectives on virtual reality training will be reviewed in three focus groups comprising four to six occupational therapists and physiotherapists. Discussion: The interviews will help to gain a deeper understanding of the phenomena under investigation to provide sound recommendations for the implementation of the virtual reality training system for routine use in neurorehabilitation complementing the quantitative clinical assessments. Trial registration: Cliniclatrials.gov Identifier: NCT01774669 (
Annals of Physical and Rehabilitation Medicine, 2014
Information technologies for rehabilitation / Annals of Physical and Rehabilitation Medicine 57S ... more Information technologies for rehabilitation / Annals of Physical and Rehabilitation Medicine 57S (2014) e84-e86 e85 motion increased from 11.3 ± 9.5 to 16 ± 10.1 (P = 0.04). Participants reported high satisfaction with the program. Discussion.-The developed telerehabilitation program is an accessible alternative to in-hospital and outpatient training and allows high-tech rehabilitation at any location.
Autonomous learning systems of significant complexity often consist of several interacting module... more Autonomous learning systems of significant complexity often consist of several interacting modules or agents. These modules collaborate to produce a system which, when viewed as a whole, exhibit behaviour that can be interpreted in some way as learning. In designing these systems, the complexity of the interactions of large numbers of modules can become overwhelming, making debugging difficult and obscuring the workings of the system when viewed from an architectural level. A way of controlling system complexity called the Layered Learning System architecture (LLS) has been developed, which offers the advantages of incremental development and testing, easier debugging and progressive upgrading facilitation. A hexapod robot has been implemented using LLS principles, with the main learning task being that of learning to walk as fast as possible without falling over.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Human pose estimation has dramatically improved thanks to the continuous developments in deep lea... more Human pose estimation has dramatically improved thanks to the continuous developments in deep learning. However, marker-free human pose estimation based on standard frame-based cameras is still slow and power hungry for real-time feedback interaction because of the huge number of operations necessary for large Convolutional Neural Network (CNN) inference. Event-based cameras such as the Dynamic Vision Sensor (DVS) quickly output sparse moving-edge information. Their sparse and rapid output is ideal for driving low-latency CNNs, thus potentially allowing real-time interaction for human pose estima-tors. Although the application of CNNs to standard frame-based cameras for human pose estimation is well established , their application to event-based cameras is still under study. This paper proposes a novel benchmark dataset of human body movements, the Dynamic Vision Sensor Human Pose dataset (DHP19). It consists of recordings from 4 synchronized 346x260 pixel DVS cameras, for a set of 33 movements with 17 subjects. DHP19 also includes a 3D pose estimation model that achieves an average 3D pose estimation error of about 8 cm, despite the sparse and reduced input data from the DVS.
Virtual reality-based training has found increasing use in neurorehabilitation to improve upper l... more Virtual reality-based training has found increasing use in neurorehabilitation to improve upper limb training and facilitate motor recovery. The aim of this study was to directly compare virtual reality-based training with conventional therapy. In a multi-center, parallel-group randomized controlled trial, patients at least 6 months after stroke onset were allocated either to an experimental group (virtual reality-based training) or a control group receiving conventional therapy (16x45 minutes within 4 weeks). The virtual reality-based training system replicated patients´ upper limb movements in real-time to manipulate virtual objects. Blinded assessors tested patients twice before, once during, and twice after the intervention up to 2-month follow-up for dexterity (primary outcome: Box and Block Test), bimanual upper limb function (Chedoke-McMaster Arm and Hand Activity Inventory), and subjective perceived changes (Stroke Impact Scale). 54 eligible patients (70 screened) participat...
Biological neural systems must grow their own connections and maintain topological relations betw... more Biological neural systems must grow their own connections and maintain topological relations between elements that are related to the sensory input surface. Artificial systems have traditionally prewired such maps, but the sensor arrangement is not always known and can be expensive to specify before run time. Here we present a method for learning and updating topographic maps in systems comprising modular, event-based elements. Using an unsupervised neural spike-timing-based learning rule combined with Hebbian learning, our algorithm uses the spatiotemporal coherence of the external world to train its network. It improves on existing algorithms by not assuming a known topography of the target map and includes a novel method for automatically detecting edge elements. We show how, for stimuli that are small relative to the sensor resolution, the temporal learning window parameters can be determined without using any user-specified constants. For stimuli that are larger relative to the sensor resolution, we provide a parameter extraction method that generally outperforms the small-stimulus method but requires one user-specified constant. The algorithm was tested on real data from a 64 × 64-pixel section of an event-based temporal contrast silicon retina and a 360-tile tactile luminous floor. It learned 95.8% of the correct neighborhood relations for the silicon retina within about 400 seconds of real-world input from a driving scene and 98.1% correct for the sensory floor after about 160 minutes of human pedestrian traffic. Residual errors occurred in regions receiving little or ambiguous input, and the learned topological representations were able to update automatically in response to simulated damage. Our algorithm has applications in the design of modular autonomous systems in which the interfaces between components are learned during operation rather than at design time.
Human pose estimation has dramatically improved thanks to the continuous developments in deep lea... more Human pose estimation has dramatically improved thanks to the continuous developments in deep learning. However, marker-free human pose estimation based on standard frame-based cameras is still slow and power hun-gry for real-time feedback interaction because of the huge number of operations necessary for large Convolutional Neural Network (CNN) inference. Event-based cameras such as the Dynamic Vision Sensor (DVS) quickly output sparse moving-edge information. Their sparse and rapid output is ideal for driving low-latency CNNs, thus poten-tially allowing real-time interaction for human pose estima-tors. Although the application of CNNs to standard frame-based cameras for human pose estimation is well estab-lished, their application to event-based cameras is still un-der study. This paper proposes a novel benchmark dataset of human body movements, the Dynamic Vision Sensor Hu-man Pose dataset (DHP19). It consists of recordings from 4 synchronized 346x260 pixel DVS cameras, for a set of 33 movements with 17 subjects. DHP19 also includes a 3D pose estimation model that achieves an average 3D pose estimation error of about 8 cm, despite the sparse and re-duced input data from the DVS.
We conceived Ada: Intelligent Space exhibit as an artificial organism, integrating a large number... more We conceived Ada: Intelligent Space exhibit as an artificial organism, integrating a large number of sensory modalities, and let it interact with visitors using a multitude of effector systems. Ada used a language of sound and light to communicate its moods, emotions, and behaviors. Here we describe the mechanisms behind Ada's sound communication, its real-time performance, and its interpretation by human subjects.
This paper and the accompanying video describe Ada, an immersive 160m 2 space for human-machine i... more This paper and the accompanying video describe Ada, an immersive 160m 2 space for human-machine interaction. Ada is conceived as an artificial organism that plays with its visitors, with multi-modal sensors consisting of pressure-sensitive floor tiles, microphones for localising and recognising sounds, and pan-tilt video cameras called gazers. Her output modalities include a 360º video projection screen capable of showing realtime computer graphics combined with multiple live video streams, RGB neon lights in the floor tiles, pan-tilt lights and an automatic neural music composition system. Visitors provided input to the space primarily by moving around, clapping their hands and speaking. Ada was presented at the Swiss national exhibition Expo.02 in Neuchâtel from May 15 to October 20, 2002, and attracted 553,700 visitors. In this paper we give a brief overview of the Ada exhibit, and describe the accompanying video.
Fast neuromorphic event-based vision sensors (Dynamic Vision Sensor, DVS) can be combined with sl... more Fast neuromorphic event-based vision sensors (Dynamic Vision Sensor, DVS) can be combined with slower conventional frame-based sensors to enable higher-quality inter-frame interpolation than traditional methods relying on fixed motion approximations using e.g. optical flow. In this work we present a new, advanced event simulator that can produce realistic scenes recorded by a camera rig with an arbitrary number of sensors located at fixed offsets. It includes a new configurable frame-based image sensor model with realistic image quality reduction effects, and an extended DVS model with more accurate characteristics. We use our simulator to train a novel reconstruction model designed for end-to-end reconstruction of high-fps video. Unlike previously published methods, our method does not require the frame and DVS cameras to have the same optics, positions, or camera resolutions. It is also not limited to objects a fixed distance from the sensor. We show that data generated by our simulator can be used to train our new model, leading to reconstructed images on public datasets of equivalent or better quality than the state of the art. We also show our sensor generalizing to data recorded by real sensors.
This paper describes a mathematical formulation for measuring how one system can estimate the con... more This paper describes a mathematical formulation for measuring how one system can estimate the consciousness of another. This consciousness estimate is always relative to the observer. The paper shows how this formulation leads to simple resolutions of some key problems of consciousness.
Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems, 2018
EVA 1 is describing a new class of emotion-aware autonomous systems delivering intelligent person... more EVA 1 is describing a new class of emotion-aware autonomous systems delivering intelligent personal assistant functionalities. EVA requires a multidisciplinary approach, combining a number of critical building blocks into a cybernetics systems/software architecture: emotion aware systems and algorithms, multimodal interaction design, cognitive modelling, decision making and recommender systems, emotion sensing as feedback for learning, and distributed (edge) computing delivering cognitive services.
Proceedings Fourth Annual Conference on Mechatronics and Machine Vision in Practice
A unique distributed cont developed to allow walk. It models competing directi robot to walk as q... more A unique distributed cont developed to allow walk. It models competing directi robot to walk as quick over given minimal in architecture features arrangement of cooper controller. This h conventional control scope for incremental facilitate accurate Accelerated learning robot's intrinsic knowledg infer additional info This was found to improve the rob particularly at the s information was limited. These concepts can be applied to autonomous robots for use in deep sea and other inhospitable environments.
Background: In recent years, virtual reality has been introduced to neurorehabilitation, in parti... more Background: In recent years, virtual reality has been introduced to neurorehabilitation, in particular with the intention of improving upper-limb training options and facilitating motor function recovery. Methods/Design: The proposed study incorporates a quantitative part and a qualitative part, termed a mixed-methods approach: (1) a quantitative investigation of the efficacy of virtual reality training compared to conventional therapy in upper-limb motor function are investigated, (2a) a qualitative investigation of patients' experiences and expectations of virtual reality training and (2b) a qualitative investigation of therapists' experiences using the virtual reality training system in the therapy setting. At three participating clinics, 60 patients at least 6 months after stroke onset will be randomly allocated to an experimental virtual reality group (EG) or to a control group that will receive conventional physiotherapy or occupational therapy (16 sessions, 45 minutes each, over the course of 4 weeks). Using custom data gloves, patients' finger and arm movements will be displayed in real time on a monitor, and they will move and manipulate objects in various virtual environments. A blinded assessor will test patients' motor and cognitive performance twice before, once during, and twice after the 4-week intervention. The primary outcome measure is the Box and Block Test. Secondary outcome measures are the Chedoke-McMaster Stroke Assessments (hand, arm and shoulder pain subscales), the Chedoke-McMaster Arm and Hand Activity Inventory, the Line Bisection Test, the Stroke Impact Scale, the MiniMentalState Examination and the Extended Barthel Index. Semistructured face-to-face interviews will be conducted with patients in the EG after intervention finalization with a focus on the patients' expectations and experiences regarding the virtual reality training. Therapists' perspectives on virtual reality training will be reviewed in three focus groups comprising four to six occupational therapists and physiotherapists. Discussion: The interviews will help to gain a deeper understanding of the phenomena under investigation to provide sound recommendations for the implementation of the virtual reality training system for routine use in neurorehabilitation complementing the quantitative clinical assessments. Trial registration: Cliniclatrials.gov Identifier: NCT01774669 (
Annals of Physical and Rehabilitation Medicine, 2014
Information technologies for rehabilitation / Annals of Physical and Rehabilitation Medicine 57S ... more Information technologies for rehabilitation / Annals of Physical and Rehabilitation Medicine 57S (2014) e84-e86 e85 motion increased from 11.3 ± 9.5 to 16 ± 10.1 (P = 0.04). Participants reported high satisfaction with the program. Discussion.-The developed telerehabilitation program is an accessible alternative to in-hospital and outpatient training and allows high-tech rehabilitation at any location.
Autonomous learning systems of significant complexity often consist of several interacting module... more Autonomous learning systems of significant complexity often consist of several interacting modules or agents. These modules collaborate to produce a system which, when viewed as a whole, exhibit behaviour that can be interpreted in some way as learning. In designing these systems, the complexity of the interactions of large numbers of modules can become overwhelming, making debugging difficult and obscuring the workings of the system when viewed from an architectural level. A way of controlling system complexity called the Layered Learning System architecture (LLS) has been developed, which offers the advantages of incremental development and testing, easier debugging and progressive upgrading facilitation. A hexapod robot has been implemented using LLS principles, with the main learning task being that of learning to walk as fast as possible without falling over.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Human pose estimation has dramatically improved thanks to the continuous developments in deep lea... more Human pose estimation has dramatically improved thanks to the continuous developments in deep learning. However, marker-free human pose estimation based on standard frame-based cameras is still slow and power hungry for real-time feedback interaction because of the huge number of operations necessary for large Convolutional Neural Network (CNN) inference. Event-based cameras such as the Dynamic Vision Sensor (DVS) quickly output sparse moving-edge information. Their sparse and rapid output is ideal for driving low-latency CNNs, thus potentially allowing real-time interaction for human pose estima-tors. Although the application of CNNs to standard frame-based cameras for human pose estimation is well established , their application to event-based cameras is still under study. This paper proposes a novel benchmark dataset of human body movements, the Dynamic Vision Sensor Human Pose dataset (DHP19). It consists of recordings from 4 synchronized 346x260 pixel DVS cameras, for a set of 33 movements with 17 subjects. DHP19 also includes a 3D pose estimation model that achieves an average 3D pose estimation error of about 8 cm, despite the sparse and reduced input data from the DVS.
Virtual reality-based training has found increasing use in neurorehabilitation to improve upper l... more Virtual reality-based training has found increasing use in neurorehabilitation to improve upper limb training and facilitate motor recovery. The aim of this study was to directly compare virtual reality-based training with conventional therapy. In a multi-center, parallel-group randomized controlled trial, patients at least 6 months after stroke onset were allocated either to an experimental group (virtual reality-based training) or a control group receiving conventional therapy (16x45 minutes within 4 weeks). The virtual reality-based training system replicated patients´ upper limb movements in real-time to manipulate virtual objects. Blinded assessors tested patients twice before, once during, and twice after the intervention up to 2-month follow-up for dexterity (primary outcome: Box and Block Test), bimanual upper limb function (Chedoke-McMaster Arm and Hand Activity Inventory), and subjective perceived changes (Stroke Impact Scale). 54 eligible patients (70 screened) participat...
Uploads
Papers by Kynan Eng