Publications

RSS feed available.

2019

  • N. Bellotto, S. Cosar, and Z. Yan, “Human detection and tracking,” in Encyclopedia of robotics, M. H. Ang, O. Khatib, and B. Siciliano, Eds., Springer, 2019.
    [BibTeX] [Abstract] [Download PDF]

    In robotics, detecting and tracking moving objects is key to implementing useful and safe robot behaviours. Identifying which of the detected objects are humans is particularly important for domestic and public environments. Typically the robot is required to collect environmental data of the surrounding area using its on-board sensors, estimating where humans are and where they are going to. Moreover, robots should detect and track humans accurately and as early as possible in order to have enough time to react accordingly

    @incollection{lirolem30916,
    title = {Human detection and tracking},
    publisher = {Springer},
    month = {July},
    editor = {M. H. Ang and O. Khatib and B. Siciliano},
    author = {Nicola Bellotto and Serhan Cosar and Zhi Yan},
    year = {2019},
    booktitle = {Encyclopedia of robotics},
    abstract = {In robotics, detecting and tracking moving objects is key to implementing useful and safe robot behaviours. Identifying which of the detected objects are humans is particularly important for domestic and public environments.
    Typically the robot is required to collect environmental data of the surrounding area using its on-board sensors, estimating where humans are and where they are going to. Moreover, robots should detect and track humans accurately and as early as possible in order to have enough time to react accordingly},
    url = {http://eprints.lincoln.ac.uk/30916/},
    keywords = {ARRAY(0x55e8c59cbb00)}
    }
  • C. Coppola, S. Cosar, D. R. Faria, and N. Bellotto, “Social activity recognition on continuous rgb-d video sequences,” International journal of social robotics, 2019.
    [BibTeX] [Abstract] [Download PDF]

    Modern service robots are provided with one or more sensors, often including RGB-D cameras, to perceive objects and humans in the environment. This paper proposes a new system for the recognition of human social activities from a continuous stream of RGB-D data. Many of the works until now have succeeded in recognising activities from clipped videos in datasets, but for robotic applications it is important to be able to move to more realistic scenarios in which such activities are not manually selected. For this reason, it is useful to detect the time intervals when humans are performing social activities, the recognition of which can contribute to trigger human-robot interactions or to detect situations of potential danger. The main contributions of this research work include a novel system for the recognition of social activities from continuous RGB-D data, combining temporal segmentation and classification, as well as a model for learning the proximity-based priors of the social activities. A new public dataset with RGB-D videos of social and individual activities is also provided and used for evaluating the proposed solutions. The results show the good performance of the system in recognising social activities from continuous RGB-D data.

    @article{lirolem35151,
    title = {Social Activity Recognition on Continuous RGB-D Video Sequences},
    publisher = {Springer},
    author = {Claudio Coppola and Serhan Cosar and Diego R. Faria and Nicola Bellotto},
    year = {2019},
    journal = {International Journal of Social Robotics},
    keywords = {ARRAY(0x55e8c5ca34d8)},
    url = {http://eprints.lincoln.ac.uk/35151/},
    abstract = {Modern service robots are provided with one or more sensors, often including RGB-D cameras, to perceive objects and humans in the environment. This paper proposes a new system for the recognition of human social activities from a continuous stream of RGB-D data. Many of the works until now have succeeded in recognising activities from clipped videos in datasets, but for robotic applications it is important to be able to move to more realistic scenarios in which such activities are not manually selected. For this reason, it is useful to detect the time intervals when humans are performing social activities, the recognition of which can contribute to trigger human-robot interactions or to detect situations of potential danger. The main contributions of this research work include a novel system for the recognition of social activities from continuous RGB-D data, combining temporal segmentation and classification, as well as a model for learning the proximity-based priors of the social activities. A new public dataset with RGB-D videos of social and individual activities is also provided and used for evaluating the proposed solutions. The results show the good performance of the system in recognising social activities from continuous RGB-D data.}
    }
  • H. Cuayahuitl, “A data-efficient deep learning approach for deployable multimodal social robots,” Neurocomputing, 2019.
    [BibTeX] [Abstract] [Download PDF]

    The deep supervised and reinforcement learning paradigms (among others) have the potential to endow interactive multimodal social robots with the ability of acquiring skills autonomously. But it is still not very clear yet how they can be best deployed in real world applications. As a step in this direction, we propose a deep learning-based approach for efficiently training a humanoid robot to play multimodal games–-and use the game of `Noughts {$\backslash$}& Crosses’ with two variants as a case study. Its minimum requirements for learning to perceive and interact are based on a few hundred example images, a few example multimodal dialogues and physical demonstrations of robot manipulation, and automatic simulations. In addition, we propose novel algorithms for robust visual game tracking and for competitive policy learning with high winning rates, which substantially outperform DQN-based baselines. While an automatic evaluation shows evidence that the proposed approach can be easily extended to new games with competitive robot behaviours, a human evaluation with 130 humans playing with the \{{$\backslash$}it Pepper\} robot confirms that highly accurate visual perception is required for successful game play.

    @article{lirolem33533,
    title = {A Data-Efficient Deep Learning Approach for Deployable Multimodal Social Robots},
    publisher = {Elsevier},
    note = {The final published version of this article can be accessed online at https://www.journals.elsevier.com/neurocomputing/},
    author = {Heriberto Cuayahuitl},
    year = {2019},
    journal = {Neurocomputing},
    url = {http://eprints.lincoln.ac.uk/33533/},
    keywords = {ARRAY(0x55e8c5b449e0)},
    abstract = {The deep supervised and reinforcement learning paradigms (among others) have the potential to endow interactive multimodal social robots with the ability of acquiring skills autonomously. But it is still not very clear yet how they can be best deployed in real world applications. As a step in this direction, we propose a deep learning-based approach for efficiently training a humanoid robot to play multimodal games---and use the game of `Noughts {$\backslash$}\& Crosses' with two variants as a case study. Its minimum requirements for learning to perceive and interact are based on a few hundred example images, a few example multimodal dialogues and physical demonstrations of robot manipulation, and automatic simulations. In addition, we propose novel algorithms for robust visual game tracking and for competitive policy learning with high winning rates, which substantially outperform DQN-based baselines. While an automatic evaluation shows evidence that the proposed approach can be easily extended to new games with competitive robot behaviours, a human evaluation with 130 humans playing with the \{{$\backslash$}it Pepper\} robot confirms that highly accurate visual perception is required for successful game play.}
    }
  • K. Elgeneidy, P. Lightbody, S. Pearson, and G. Neumann, “Characterising 3d-printed soft fin ray robotic fingers with layer jamming capability for delicate grasping,” in Robosoft 2019, 2019.
    [BibTeX] [Abstract] [Download PDF]

    Motivated by the growing need within the agrifood industry to automate the handling of delicate produce, this paper presents soft robotic fingers utilising the Fin Ray effect to passively and gently adapt to delicate targets. The proposed Soft Fin Ray fingers feature thin ribs and are entirely 3D printed from a flexible material (NinjaFlex) to enhance their shape adaptation, compared to the original Fin Ray fingers. To overcome their reduced force generation, the effects of the angle and spacing of the flexible ribs were experimentally characterised. The results showed that at large displacements, layer jamming between tilted flexible ribs can significantly enhance the force generation, while minimal contact forces can be still maintained at small displacements for delicate grasping.

    @inproceedings{lirolem34950,
    month = {June},
    title = {Characterising 3D-printed Soft Fin Ray Robotic Fingers with Layer Jamming Capability for Delicate Grasping},
    booktitle = {RoboSoft 2019},
    year = {2019},
    author = {Khaled Elgeneidy and Peter Lightbody and Simon Pearson and Gerhard Neumann},
    abstract = {Motivated by the growing need within the agrifood industry to automate the handling of delicate produce, this paper presents soft robotic fingers utilising the Fin Ray effect to passively and gently adapt to delicate targets. The proposed Soft Fin Ray fingers feature thin ribs and are entirely 3D printed from a flexible material (NinjaFlex) to enhance their shape adaptation, compared to the original Fin Ray fingers. To overcome their reduced force generation, the effects of
    the angle and spacing of the flexible ribs were experimentally characterised. The results showed that at large displacements, layer jamming between tilted flexible ribs can significantly enhance the force generation, while minimal contact forces can be still maintained at small displacements for delicate grasping.},
    url = {http://eprints.lincoln.ac.uk/34950/},
    keywords = {ARRAY(0x55e8c5c60b48)}
    }
  • K. Elgeneidy, G. Neumann, S. Pearson, M. Jackson, and N. Lohse, “Contact detection and size estimation using a modular soft gripper with embedded flex sensors,” in International conference on intelligent robots and systems (iros 2018), 2019.
    [BibTeX] [Abstract] [Download PDF]

    Grippers made from soft elastomers are able to passively and gently adapt to their targets allowing deformable objects to be grasped safely without causing bruise or damage. However, it is difficult to regulate the contact forces due to the lack of contact feedback for such grippers. In this paper, a modular soft gripper is presented utilizing interchangeable soft pneumatic actuators with embedded flex sensors as fingers of the gripper. The fingers can be assembled in different configurations using 3D printed connectors. The paper investigates the potential of utilizing the simple sensory feedback from the flex and pressure sensors to make additional meaningful inferences regarding the contact state and grasped object size. We study the effect of the grasped object size and contact type on the combined feedback from the embedded flex sensors of opposing fingers. Our results show that a simple linear relationship exists between the grasped object size and the final flex sensor reading at fixed input conditions, despite the variation in object weight and contact type. Additionally, by simply monitoring the time series response from the flex sensor, contact can be detected by comparing the response to the known free-bending response at the same input conditions. Furthermore, by utilizing the measured internal pressure supplied to the soft fingers, it is possible to distinguish between power and pinch grasps, as the contact type affects the rate of change in the flex sensor readings against the internal pressure.

    @inproceedings{lirolem34713,
    title = {Contact Detection and Size Estimation Using a Modular Soft Gripper with Embedded Flex Sensors},
    month = {January},
    year = {2019},
    author = {Khaled Elgeneidy and Gerhard Neumann and Simon Pearson and Michael Jackson and Niels Lohse},
    booktitle = {International Conference on Intelligent Robots and Systems (IROS 2018)},
    abstract = {Grippers made from soft elastomers are able to passively and gently adapt to their targets allowing deformable objects to be grasped safely without causing bruise or damage. However, it is difficult to regulate the contact forces due to the lack of contact feedback for such grippers. In this paper, a modular soft gripper is presented utilizing interchangeable soft pneumatic actuators with embedded flex sensors as fingers of the gripper. The fingers can be assembled in different configurations using 3D printed connectors. The paper investigates the potential of utilizing the simple sensory feedback from the flex and pressure sensors to make additional meaningful inferences regarding the contact state and grasped object size. We study the effect of the grasped object size and contact type on the combined feedback from the embedded flex sensors of opposing fingers. Our results show that a simple linear relationship exists between the grasped object size and the final flex sensor reading at fixed input conditions, despite the variation in object weight and contact type. Additionally, by simply monitoring the time series response from the flex sensor, contact can be detected by comparing the response to the known free-bending response at the same input conditions. Furthermore, by utilizing the measured internal pressure supplied to the soft fingers, it is possible to distinguish between power and pinch grasps, as the contact type affects the rate of change in the flex sensor readings against the internal pressure.},
    keywords = {ARRAY(0x55e8c59cddd0)},
    url = {http://eprints.lincoln.ac.uk/34713/}
    }
  • C. Fox, Use and citation of paper “fox et al (2018), ?when should the chicken cross the road? game theory for autonomous vehicle – human interactions conference paper?” by the law commission to review and potentially change the law of the uk on autonomous vehicles. cited in their consultation report, “automated vehicles: a joint preliminary consultation paper” on p174, ref 651., 2019.
    [BibTeX] [Abstract] [Download PDF]

    Topic of this consultation: The Centre for Connected and Automated Vehicles (CCAV) has asked the Law Commission of England and Wales and the Scottish Law Commission to examine options for regulating automated road vehicles. It is a three-year project, running from March 2018 to March 2021. This preliminary consultation paper focuses on the safety of passenger vehicles. Driving automation refers to a broad range of vehicle technologies. Examples range from widely-used technologies that assist human drivers (such as cruise control) to vehicles that drive themselves with no human intervention. We concentrate on automated driving systems which do not need human drivers for at least part of the journey. This paper looks at are three key themes. First, we consider how safety can be assured before and after automated driving systems are deployed. Secondly, we explore criminal and civil liability. Finally, we examine the need to adapt road rules for artificial intelligence.

    @misc{lirolem34922,
    title = {Use and citation of paper "Fox et al (2018), ?When should the chicken cross the road? Game theory for autonomous vehicle - human interactions conference paper?" by the Law Commission to review and potentially change the law of the UK on autonomous vehicles. Cited in their consultation report, "Automated Vehicles: A joint preliminary consultation paper" on p174, ref 651.},
    month = {January},
    author = {Charles Fox},
    year = {2019},
    journal = {Automated Vehicles: A joint preliminary consultation paper},
    url = {http://eprints.lincoln.ac.uk/34922/},
    keywords = {ARRAY(0x55e8c59b9760)},
    abstract = {Topic of this consultation: The Centre for Connected and Automated Vehicles (CCAV) has
    asked the Law Commission of England and Wales and the Scottish Law Commission to
    examine options for regulating automated road vehicles. It is a three-year project, running from
    March 2018 to March 2021. This preliminary consultation paper focuses on the safety of
    passenger vehicles.
    Driving automation refers to a broad range of vehicle technologies. Examples range from
    widely-used technologies that assist human drivers (such as cruise control) to vehicles that
    drive themselves with no human intervention. We concentrate on automated driving systems
    which do not need human drivers for at least part of the journey.
    This paper looks at are three key themes. First, we consider how safety can be assured before
    and after automated driving systems are deployed. Secondly, we explore criminal and civil
    liability. Finally, we examine the need to adapt road rules for artificial intelligence.}
    }
  • S. Kottayil, P. Tsoleridis, K. Rossa, and C. Fox, “Investigation of driver route choice behaviour using bluetooth data,” in 15th world conference on transport research, 2019.
    [BibTeX] [Abstract] [Download PDF]

    Many local authorities use small-scale transport models to manage their transportation networks. These may assume drivers? behaviour to be rational in choosing the fastest route, and thus that all drivers behave the same given an origin and destination, leading to simplified aggregate flow models, fitted to anonymous traffic flow measurements. Recent price falls in traffic sensors, data storage, and compute power now enable Data Science to empirically test such assumptions, by using per-driver data to infer route selection from sensor observations and compare with optimal route selection. A methodology is presented using per-driver data to analyse driver route choice behaviour in transportation networks. Traffic flows on multiple measurable routes for origin destination pairs are compared based on the length of each route. A driver rationality index is defined by considering the shortest physical route between an origin-destination pair. The proposed method is intended to aid calibration of parameters used in traffic assignment models e.g. weights in generalized cost formulations or dispersion within stochastic user equilibrium models. The method is demonstrated using raw sensor datasets collected through Bluetooth sensors in the area of Chesterfield, Derbyshire, UK. The results for this region show that routes with a significant difference in lengths of their paths have the majority (71\%) of drivers using the optimal path but as the difference in length decreases, the probability of suboptimal route choice decreases (27\%). The methodology can be used for extended research considering the impact on route choice of other factors including travel time and road specific conditions.

    @inproceedings{lirolem34791,
    booktitle = {15th World Conference on Transport Research},
    author = {Sreedevi Kottayil and Panagiotis Tsoleridis and Kacper Rossa and Charles Fox},
    year = {2019},
    publisher = {Elsevier},
    month = {May},
    title = {Investigation of Driver Route Choice Behaviour using Bluetooth Data},
    abstract = {Many local authorities use small-scale transport models to manage their transportation networks. These may assume drivers? behaviour to be rational in choosing the fastest route, and thus that all drivers behave the same given an origin and destination, leading to simplified aggregate flow models, fitted to anonymous traffic flow measurements. Recent price falls in traffic sensors, data storage, and compute power now enable Data Science to empirically test such assumptions, by using per-driver data to infer route selection from sensor observations and compare with optimal route selection. A methodology is presented using per-driver data to analyse driver route choice behaviour in transportation networks. Traffic flows on multiple measurable routes for origin destination pairs are compared based on the length of each route. A driver rationality index is defined by considering the shortest physical route between an origin-destination pair. The proposed method is intended to aid calibration of parameters used in traffic assignment models e.g. weights in generalized cost formulations or dispersion within stochastic user equilibrium models. The method is demonstrated using raw sensor datasets collected through Bluetooth sensors in the area of Chesterfield, Derbyshire, UK. The results for this region show that routes with a significant difference in lengths of their paths have the majority (71\%) of drivers using the optimal path but as the difference in length decreases, the probability of suboptimal route choice decreases (27\%). The methodology can be used for extended research considering the impact on route choice of other factors including travel time and road specific conditions.},
    keywords = {ARRAY(0x55e8c59c7608)},
    url = {http://eprints.lincoln.ac.uk/34791/}
    }
  • J. Lock, G. Cielniak, and N. Bellotto, “Active object search with a mobile device for people with visual impairments,” in 14th international conference on computer vision theory and applications (visapp), 2019.
    [BibTeX] [Abstract] [Download PDF]

    Modern smartphones can provide a multitude of services to assist people with visual impairments, and their cameras in particular can be useful for assisting with tasks, such as reading signs or searching for objects in unknown environments. Previous research has looked at ways to solve these problems by processing the camera’s video feed, but very little work has been done in actively guiding the user towards specific points of interest, maximising the effectiveness of the underlying visual algorithms. In this paper, we propose a control algorithm based on a Markov Decision Process that uses a smartphone?s camera to generate real-time instructions to guide a user towards a target object. The solution is part of a more general active vision application for people with visual impairments. An initial implementation of the system on a smartphone was experimentally evaluated with participants with healthy eyesight to determine the performance of the control algorithm. The results show the effectiveness of our solution and its potential application to help people with visual impairments find objects in unknown environments.

    @inproceedings{lirolem34596,
    author = {Jacobus Lock and Grzegorz Cielniak and Nicola Bellotto},
    year = {2019},
    booktitle = {14th International Conference on Computer Vision Theory and Applications (VISAPP)},
    title = {Active Object Search with a Mobile Device for People with Visual Impairments},
    abstract = {Modern smartphones can provide a multitude of services to assist people with visual impairments, and their cameras in particular can be useful for assisting with tasks, such as reading signs or searching for objects in unknown environments. Previous research has looked at ways to solve these problems by processing the camera's video feed, but very little work has been done in actively guiding the user towards specific points of interest, maximising the effectiveness of the underlying visual algorithms. In this paper, we propose a control algorithm based on a Markov Decision Process that uses a smartphone?s camera to generate real-time instructions to guide a user towards a target object. The solution is part of a more general active vision application for people with visual impairments. An initial implementation of the system on a smartphone was experimentally evaluated with participants with healthy eyesight to determine the performance of the control algorithm. The results show the effectiveness of our solution and its potential application to help people with visual impairments find objects in unknown environments.},
    keywords = {ARRAY(0x55e8c59976e0)},
    url = {http://eprints.lincoln.ac.uk/34596/}
    }

2018

  • P. Bosilj, T. Duckett, and G. Cielniak, “Analysis of morphology-based features for classification of crop and weeds in precision agriculture,” Ieee robotics and automation letters, vol. 3, iss. 4, p. 2950–2956, 2018.
    [BibTeX] [Abstract] [Download PDF]

    Determining the types of vegetation present in an image is a core step in many precision agriculture tasks. In this paper, we focus on pixel-based approaches for classification of crops versus weeds, especially for complex cases involving overlapping plants and partial occlusion. We examine the benefits of multi-scale and content-driven morphology-based descriptors called Attribute Profiles. These are compared to state-of-the art keypoint descriptors with a fixed neighbourhood previously used in precision agriculture, namely Histograms of Oriented Gradients and Local Binary Patterns. The proposed classification technique is especially advantageous when coupled with morphology-based segmentation on a max-tree structure, as the same representation can be re-used for feature extraction. The robustness of the approach is demonstrated by an experimental evaluation on two datasets with different crop types. The proposed approach compared favourably to state-of-the-art approaches without an increase in computational complexity, while being able to provide descriptors at a higher resolution.

    @article{lirolem32371,
    title = {Analysis of morphology-based features for classification of crop and weeds in precision agriculture},
    number = {4},
    pages = {2950--2956},
    author = {Petra Bosilj and Tom Duckett and Grzegorz Cielniak},
    year = {2018},
    month = {October},
    publisher = {IEEE},
    journal = {IEEE Robotics and Automation Letters},
    volume = {3},
    keywords = {ARRAY(0x55e8c5b7bfa0)},
    url = {http://eprints.lincoln.ac.uk/32371/},
    abstract = {Determining the types of vegetation present in an image is a core step in many precision agriculture tasks. In this paper, we focus on pixel-based approaches for classification of crops versus weeds, especially for complex cases involving overlapping plants and partial occlusion. We examine the benefits of multi-scale and content-driven morphology-based descriptors called Attribute Profiles. These are compared to state-of-the art keypoint descriptors with a fixed neighbourhood previously used in precision agriculture, namely Histograms of Oriented Gradients and Local Binary Patterns. The proposed classification technique is especially advantageous when coupled with morphology-based segmentation on a max-tree structure, as the same representation can be re-used for feature extraction. The robustness of the approach is demonstrated by an experimental evaluation on two datasets with different crop types. The proposed approach compared favourably to state-of-the-art approaches without an increase in computational complexity, while being able to provide descriptors at a higher resolution.}
    }
  • F. Camara, O. Giles, M. Rothmuller, P. Rasmussen, A. Vendelbo-Larsen, G. Markkula, Y-M. Lee, N. Merat, and C. Fox, “Predicting pedestrian road-crossing assertiveness for autonomous vehicle control,” in 21st ieee international conference on intelligent transportation systems, 2018.
    [BibTeX] [Abstract] [Download PDF]

    Autonomous vehicles (AVs) must interact with other road users including pedestrians. Unlike passive environments, pedestrians are active agents having their own utilities and decisions, which must be inferred and predicted by AVs in order to control interactions with them and navigation around them. In particular, when a pedestrian wishes to cross the road in front of the vehicle at an unmarked crossing, the pedestrian and AV must compete for the space, which may be considered as a game-theoretic interaction in which one agent must yield to the other. To inform AV controllers in this setting, this study collects and analyses data from real-world human road crossings to determine what features of crossing behaviours are predictive about the level of assertiveness of pedestrians and of the eventual winner of the interactions. It presents the largest and most detailed data set of its kind known to us, and new methods to analyze and predict pedestrian-vehicle interactions based upon it. Pedestrian-vehicle interactions are decomposed into sequences of independent discrete events. We use probabilistic methods ? regression and decision tree regression ? and sequence analysis to analyze sets and sub-sequences of actions used by both pedestrians and human drivers while crossing at an intersection, to find common patterns of behaviour and to predict the winner of each interaction. We report on the particular features found to be predictive and which can thus be integrated into game- theoretic AV controllers to inform real-time interactions.

    @inproceedings{lirolem33089,
    year = {2018},
    author = {F Camara and O Giles and M Rothmuller and PH Rasmussen and A Vendelbo-Larsen and G Markkula and Y-M Lee and N Merat and Charles Fox},
    booktitle = {21st IEEE International Conference on Intelligent Transportation Systems},
    title = {Predicting pedestrian road-crossing assertiveness for autonomous vehicle control},
    month = {November},
    publisher = {IEEE},
    keywords = {ARRAY(0x55e8c5b4c238)},
    url = {http://eprints.lincoln.ac.uk/33089/},
    abstract = {Autonomous vehicles (AVs) must interact with other
    road users including pedestrians. Unlike passive environments,
    pedestrians are active agents having their own utilities and
    decisions, which must be inferred and predicted by AVs in order
    to control interactions with them and navigation around them.
    In particular, when a pedestrian wishes to cross the road in
    front of the vehicle at an unmarked crossing, the pedestrian
    and AV must compete for the space, which may be considered
    as a game-theoretic interaction in which one agent must yield
    to the other. To inform AV controllers in this setting, this study
    collects and analyses data from real-world human road crossings
    to determine what features of crossing behaviours are predictive
    about the level of assertiveness of pedestrians and of the eventual
    winner of the interactions. It presents the largest and most
    detailed data set of its kind known to us, and new methods to
    analyze and predict pedestrian-vehicle interactions based upon
    it. Pedestrian-vehicle interactions are decomposed into sequences
    of independent discrete events. We use probabilistic methods ?
    regression and decision tree regression ? and sequence analysis
    to analyze sets and sub-sequences of actions used by both
    pedestrians and human drivers while crossing at an intersection,
    to find common patterns of behaviour and to predict the winner
    of each interaction. We report on the particular features found
    to be predictive and which can thus be integrated into game-
    theoretic AV controllers to inform real-time interactions.}
    }
  • H. Cuayahuitl, S. Ryu, D. Lee, and J. Kim, “A study on dialogue reward prediction for open-ended conversational agents,” in Neurips workshop on conversational ai, 2018.
    [BibTeX] [Abstract] [Download PDF]

    The amount of dialogue history to include in a conversational agent is often underestimated and/or set in an empirical and thus possibly naive way. This suggests that principled investigations into optimal context windows are urgently needed given that the amount of dialogue history and corresponding representations can play an important role in the overall performance of a conversational system. This paper studies the amount of history required by conversational agents for reliably predicting dialogue rewards. The task of dialogue reward prediction is chosen for investigating the effects of varying amounts of dialogue history and their impact on system performance. Experimental results using a dataset of 18K human-human dialogues report that lengthy dialogue histories of at least 10 sentences are preferred (25 sentences being the best in our experiments) over short ones, and that lengthy histories are useful for training dialogue reward predictors with strong positive correlations between target dialogue rewards and predicted ones.

    @inproceedings{lirolem34433,
    month = {December},
    publisher = {arXiv},
    title = {A Study on Dialogue Reward Prediction for Open-Ended Conversational Agents},
    booktitle = {NeurIPS Workshop on Conversational AI},
    author = {Heriberto Cuayahuitl and Seonghan Ryu and Donghyeon Lee and Jihie Kim},
    year = {2018},
    url = {http://eprints.lincoln.ac.uk/34433/},
    keywords = {ARRAY(0x55e8c5b5e108)},
    abstract = {The amount of dialogue history to include in a conversational agent is often underestimated and/or set in an empirical and thus possibly naive way. This suggests that principled investigations into optimal context windows are urgently needed given that the amount of dialogue history and corresponding representations can play an important role in the overall performance of a conversational system. This paper studies the amount of history required by conversational agents for reliably predicting dialogue rewards. The task of dialogue reward prediction is chosen for investigating the effects of varying amounts of dialogue history and their impact on system performance. Experimental results using a dataset of 18K human-human dialogues report that lengthy dialogue histories of at least 10 sentences are preferred (25 sentences being the best in our experiments) over short ones, and that lengthy histories are useful for training dialogue reward predictors with strong positive correlations between target dialogue rewards and predicted ones.}
    }
  • Q. Fu, C. Hu, J. Peng, and S. Yue, “Shaping the collision selectivity in a looming sensitive neuron model with parallel on and off pathways and spike frequency adaptation,” Neural networks, vol. 106, p. 127–143, 2018.
    [BibTeX] [Abstract] [Download PDF]

    Shaping the collision selectivity in vision-based artificial collision-detecting systems is still an open challenge. This paper presents a novel neuron model of a locust looming detector, i.e. the lobula giant movement detector (LGMD1), in order to provide effective solutions to enhance the collision selectivity of looming objects over other visual challenges. We propose an approach to model the biologically plausible mechanisms of ON and OFF pathways and a biophysical mechanism of spike frequency adaptation (SFA) in the proposed LGMD1 visual neural network. The ON and OFF pathways can separate both dark and light looming features for parallel spatiotemporal computations. This works effectively on perceiving a potential collision from dark or light objects that approach; such a bio-plausible structure can also separate LGMD1’s collision selectivity to its neighbouring looming detector – the LGMD2.The SFA mechanism can enhance the LGMD1’s collision selectivity to approaching objects rather than receding and translating stimuli, which is a significant improvement compared with similar LGMD1 neuron models. The proposed framework has been tested using off-line tests of synthetic and real-world stimuli, as well as on-line bio-robotic tests. The enhanced collision selectivity of the proposed model has been validated in systematic experiments. The computational simplicity and robustness of this work have also been verified by the bio-robotic tests, which demonstrates potential in building neuromorphic sensors for collision detection in both a fast and reliable manner.

    @article{lirolem31536,
    year = {2018},
    author = {Qinbing Fu and Cheng Hu and Jigen Peng and Shigang Yue},
    pages = {127--143},
    title = {Shaping the collision selectivity in a looming sensitive neuron model with parallel ON and OFF pathways and spike frequency adaptation},
    volume = {106},
    journal = {Neural Networks},
    month = {December},
    publisher = {Elsevier for European Neural Network Society (ENNS)},
    abstract = {Shaping the collision selectivity in vision-based artificial collision-detecting systems is still an open challenge. This paper presents a novel neuron model of a locust looming detector, i.e. the lobula giant movement detector (LGMD1), in order to provide effective solutions to enhance the collision selectivity of looming objects over other visual challenges. We propose an approach to model the biologically plausible mechanisms of ON and OFF pathways and a biophysical mechanism of spike frequency adaptation (SFA) in the proposed LGMD1 visual neural network. The ON and OFF pathways can separate both dark and light looming features for parallel spatiotemporal computations. This works effectively on perceiving a potential collision from dark or light objects that approach; such a bio-plausible structure can also separate LGMD1's collision selectivity to its neighbouring looming detector -- the LGMD2.The SFA mechanism can enhance the LGMD1's collision selectivity to approaching objects rather than receding and translating stimuli, which is a significant improvement compared with similar LGMD1 neuron models. The proposed framework has been tested using off-line tests of synthetic and real-world stimuli, as well as on-line bio-robotic tests. The enhanced collision selectivity of the proposed model has been validated in systematic experiments. The computational simplicity and robustness of this work have also been verified by the bio-robotic tests, which demonstrates potential in building neuromorphic sensors for collision detection in both a fast and reliable manner.},
    url = {http://eprints.lincoln.ac.uk/31536/},
    keywords = {ARRAY(0x55e8c59976b0)}
    }
  • Q. Fu, N. Bellotto, C. Hu, and S. Yue, “Performance of a visual fixation model in an autonomous micro robot inspired by drosophila physiology,” in Ieee international conference on robotics and biomimetics, 2018.
    [BibTeX] [Abstract] [Download PDF]

    In nature, lightweight and low-powered insects are ideal model systems to study motion perception strategies. Understanding the underlying characteristics and functionality of insects? visual systems is not only attractive to neural system modellers, but also critical in providing effective solutions to future robotics. This paper presents a novel modelling of dynamic vision system inspired by Drosophila physiology for mimicking fast motion tracking and a closed-loop behavioural response to ?xation. The proposed model was realised on embedded system in an autonomous micro robot which has limited computational resources. A monocular camera was applied as the only motion sensing modality. Systematic experiments including open-loop and closed-loop bio-robotic tests validated the proposed visual ?xation model: the robot showed motion tracking and ?xation behaviours similarly to insects; the image processing frequency can maintain 25 {$\sim$} 45Hz. Arena tests also demonstrated a successful following behaviour aroused by ?xation in navigation.

    @inproceedings{lirolem33846,
    title = {Performance of a Visual Fixation Model in an Autonomous Micro Robot Inspired by Drosophila Physiology},
    month = {December},
    year = {2018},
    author = {Qinbing Fu and Nicola Bellotto and Cheng Hu and Shigang Yue},
    booktitle = {IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND BIOMIMETICS},
    abstract = {In nature, lightweight and low-powered insects are ideal model systems to study motion perception strategies. Understanding the underlying characteristics and functionality of insects? visual systems is not only attractive to neural system modellers, but also critical in providing effective solutions to future robotics. This paper presents a novel modelling of dynamic vision system inspired by Drosophila physiology for mimicking fast motion tracking and a closed-loop behavioural response to ?xation. The proposed model was realised on embedded system in an autonomous micro robot which has limited computational resources. A monocular camera was applied as the only motion sensing modality. Systematic experiments including open-loop and closed-loop bio-robotic tests validated the proposed visual ?xation model: the robot showed motion tracking and ?xation behaviours similarly to insects; the image processing frequency can maintain 25 {$\sim$} 45Hz. Arena tests also demonstrated a successful following behaviour aroused by ?xation in navigation.},
    url = {http://eprints.lincoln.ac.uk/33846/},
    keywords = {ARRAY(0x55e8c5b51460)}
    }
  • H. van Hoof, G. Neumann, and J. Peters, “Non-parametric policy search with limited information loss,” Journal of machine learning research, vol. 18, iss. 73, p. 1–46, 2018.
    [BibTeX] [Abstract] [Download PDF]

    Learning complex control policies from non-linear and redundant sensory input is an important challenge for reinforcement learning algorithms. Non-parametric methods that approximate values functions or transition models can address this problem, by adapting to the complexity of the dataset. Yet, many current non-parametric approaches rely on unstable greedy maximization of approximate value functions, which might lead to poor convergence or oscillations in the policy update. A more robust policy update can be obtained by limiting the information loss between successive state-action distributions. In this paper, we develop a policy search algorithm with policy updates that are both robust and non-parametric. Our method can learn non-parametric control policies for infinite horizon continuous Markov decision processes with non-linear and redundant sensory representations. We investigate how we can use approximations of the kernel function to reduce the time requirements of the demanding non-parametric computations. In our experiments, we show the strong performance of the proposed method, and how it can be approximated effi- ciently. Finally, we show that our algorithm can learn a real-robot underpowered swing-up task directly from image data.

    @article{lirolem28020,
    number = {73},
    title = {Non-parametric policy search with limited information loss},
    author = {Herke van Hoof and Gerhard Neumann and Jan Peters},
    year = {2018},
    pages = {1--46},
    publisher = {Journal of Machine Learning Research},
    month = {December},
    volume = {18},
    journal = {Journal of Machine Learning Research},
    abstract = {Learning complex control policies from non-linear and redundant sensory input is an important
    challenge for reinforcement learning algorithms. Non-parametric methods that
    approximate values functions or transition models can address this problem, by adapting
    to the complexity of the dataset. Yet, many current non-parametric approaches rely on
    unstable greedy maximization of approximate value functions, which might lead to poor
    convergence or oscillations in the policy update. A more robust policy update can be obtained
    by limiting the information loss between successive state-action distributions. In this
    paper, we develop a policy search algorithm with policy updates that are both robust and
    non-parametric. Our method can learn non-parametric control policies for infinite horizon
    continuous Markov decision processes with non-linear and redundant sensory representations.
    We investigate how we can use approximations of the kernel function to reduce the
    time requirements of the demanding non-parametric computations. In our experiments, we
    show the strong performance of the proposed method, and how it can be approximated effi-
    ciently. Finally, we show that our algorithm can learn a real-robot underpowered swing-up
    task directly from image data.},
    url = {http://eprints.lincoln.ac.uk/28020/},
    keywords = {ARRAY(0x55e8c5b3c650)}
    }
  • S. Indurthi, S. Yu, S. Back, and H. Cuayahuitl, “Cut to the chase: a context zoom-in network for reading comprehension,” in Empirical methods in natural language processing (emnlp), 2018.
    [BibTeX] [Abstract] [Download PDF]

    In recent years many deep neural networks have been proposed to solve Reading Comprehension (RC) tasks. Most of these models suffer from reasoning over long documents and do not trivially generalize to cases where the answer is not present as a span in a given document. We present a novel neural-based architecture that is capable of extracting relevant regions based on a given question-document pair and generating a well-formed answer. To show the effectiveness of our architecture, we conducted several experiments on the recently proposed and challenging RC dataset ?NarrativeQA?. The proposed architecture outperforms state-of-the-art results (Tay et al., 2018) by 12.62\% (ROUGE-L) relative improvement.

    @inproceedings{lirolem34105,
    booktitle = {Empirical Methods in Natural Language Processing (EMNLP)},
    author = {Satish Indurthi and Seunghak Yu and Seohyun Back and Heriberto Cuayahuitl},
    year = {2018},
    publisher = {Association for Computational Linguistics},
    month = {October},
    title = {Cut to the Chase: A Context Zoom-in Network for Reading Comprehension},
    abstract = {In recent years many deep neural networks have been proposed to solve Reading Comprehension (RC) tasks. Most of these models suffer from reasoning over long documents and do not trivially generalize to cases where the answer is not present as a span in a given document. We present a novel neural-based architecture that is capable of extracting relevant regions based on a given question-document pair and generating a well-formed answer. To show the effectiveness of our architecture, we conducted several experiments on the recently proposed and challenging RC dataset ?NarrativeQA?. The proposed architecture outperforms state-of-the-art results (Tay et al., 2018) by 12.62\% (ROUGE-L) relative improvement.},
    url = {http://eprints.lincoln.ac.uk/34105/},
    keywords = {ARRAY(0x55e8c5b5fa68)}
    }
  • H. Wang, S. Yue, J. Peng, P. Baxter, C. Zhang, and Z. Wang, “A model for detection of angular velocity of image motion based on the temporal tuning of the drosophila,” in Icann 2018, 2018, p. 37–46.
    [BibTeX] [Abstract] [Download PDF]

    We propose a new bio-plausible model based on the visual systems of Drosophila for estimating angular velocity of image motion in insects? eyes. The model implements both preferred direction motion enhancement and non-preferred direction motion suppression which is discovered in Drosophila?s visual neural circuits recently to give a stronger directional selectivity. In addition, the angular velocity detecting model (AVDM) produces a response largely independent of the spatial frequency in grating experiments which enables insects to estimate the flight speed in cluttered environments. This also coincides with the behaviour experiments of honeybee flying through tunnels with stripes of different spatial frequencies.

    @inproceedings{lirolem33104,
    author = {Huatian Wang and Shigang Yue and Jigen Peng and Paul Baxter and Chun Zhang and Zhihua Wang},
    year = {2018},
    booktitle = {ICANN 2018},
    pages = {37--46},
    title = {A Model for Detection of Angular Velocity of Image Motion Based on the Temporal Tuning of the Drosophila},
    publisher = {Springer, Cham},
    month = {December},
    abstract = {We propose a new bio-plausible model based on the visual systems of Drosophila for estimating angular velocity of image motion in insects? eyes. The model implements both preferred direction motion enhancement and non-preferred direction motion suppression which is discovered in Drosophila?s visual neural circuits recently to give a stronger directional selectivity. In addition, the angular velocity detecting model (AVDM) produces a response largely independent of the spatial frequency in grating experiments which enables insects to estimate the flight speed in cluttered environments. This also coincides with the behaviour experiments of honeybee flying through tunnels with stripes of different spatial frequencies.},
    keywords = {ARRAY(0x55e8c5b55f50)},
    url = {http://eprints.lincoln.ac.uk/33104/}
    }
  • H. Wang, J. Peng, and S. Yue, “A directionally selective small target motion detecting visual neural network in cluttered backgrounds,” Ieee transaction on cybernetics, 2018.
    [BibTeX] [Abstract] [Download PDF]

    Discriminating targets moving against a cluttered background is a huge challenge, let alone detecting a target as small as one or a few pixels and tracking it in flight. In the insect’s visual system, a class of specific neurons, called small target motion detectors (STMDs), have been identified as showing exquisite selectivity for small target motion. Some of the STMDs have also demonstrated direction selectivity which means these STMDs respond strongly only to their preferred motion direction. Direction selectivity is an important property of these STMD neurons which could contribute to tracking small targets such as mates in flight. However, little has been done on systematically modeling these directionally selective STMD neurons. In this paper, we propose a directionally selective STMD-based neural network for small target detection in a cluttered background. In the proposed neural network, a new correlation mechanism is introduced for direction selectivity via correlating signals relayed from two pixels. Then, a lateral inhibition mechanism is implemented on the spatial field for size selectivity of the STMD neurons. Finally, a population vector algorithm is used to encode motion direction of small targets. Extensive experiments showed that the proposed neural network not only is in accord with current biological findings, i.e., showing directional preferences, but also worked reliably in detecting small targets against cluttered backgrounds.

    @article{lirolem33420,
    month = {October},
    publisher = {IEEE},
    note = {The final published version of this article can be accessed online at https://ieeexplore.ieee.org/document/8485659},
    title = {A Directionally Selective Small Target Motion Detecting Visual Neural Network in Cluttered Backgrounds},
    journal = {IEEE Transaction on Cybernetics},
    author = {Hongxin Wang and Jigen Peng and Shigang Yue},
    year = {2018},
    keywords = {ARRAY(0x55e8c5b6fc38)},
    url = {http://eprints.lincoln.ac.uk/33420/},
    abstract = {Discriminating targets moving against a cluttered background is a huge challenge, let alone detecting a target as small as one or a few pixels and tracking it in flight. In the insect's visual system, a class of specific neurons, called small target motion detectors (STMDs), have been identified as showing exquisite selectivity for small target motion. Some of the STMDs have also demonstrated direction selectivity which means these STMDs respond strongly only to their preferred motion direction. Direction selectivity is an important property of these STMD neurons which could contribute to tracking small targets such as mates in flight. However, little has been done on systematically modeling these directionally selective STMD neurons. In this paper, we propose a directionally selective STMD-based neural network for small target detection in a cluttered background. In the proposed neural network, a new correlation mechanism is introduced for direction selectivity via correlating signals relayed from two pixels. Then, a lateral inhibition mechanism is implemented on the spatial field for size selectivity of the STMD neurons. Finally, a population vector algorithm is used to encode motion direction of small targets. Extensive experiments showed that the proposed neural network not only is in accord with current biological findings, i.e., showing directional preferences, but also worked reliably in detecting small targets against cluttered backgrounds.}
    }
  • J. Zhao, C. Hu, C. Zhang, Z. Wang, and S. Yue, “A bio-inspired collision detector for small quadcopter,” in 2018 international joint conference on neural networks (ijcnn), 2018, p. 1–7.
    [BibTeX] [Abstract] [Download PDF]

    The sense and avoid capability enables insects to fly versatilely and robustly in dynamic and complex environment. Their biological principles are so practical and efficient that inspired we human imitating them in our flying machines. In this paper, we studied a novel bio-inspired collision detector and its application on a quadcopter. The detector is inspired from Lobula giant movement detector (LGMD) neurons in the locusts, and modeled into an STM32F407 Microcontroller Unit (MCU). Compared to other collision detecting methods applied on quadcopters, we focused on enhancing the collision accuracy in a bio-inspired way that can considerably increase the computing efficiency during an obstacle detecting task even in complex and dynamic environment. We designed the quadcopter’s responding operation to imminent collisions and tested this bio-inspired system in an indoor arena. The observed results from the experiments demonstrated that the LGMD collision detector is feasible to work as a vision module for the quadcopter’s collision avoidance task.

    @inproceedings{lirolem34847,
    pages = {1--7},
    booktitle = {2018 International Joint Conference on Neural Networks (IJCNN)},
    author = {Jiannan Zhao and Cheng Hu and Chun Zhang and Zhihua Wang and Shigang Yue},
    year = {2018},
    publisher = {IEEE},
    month = {October},
    title = {A Bio-inspired Collision Detector for Small Quadcopter},
    keywords = {ARRAY(0x55e8c5b6fc08)},
    url = {http://eprints.lincoln.ac.uk/34847/},
    abstract = {The sense and avoid capability enables insects to fly versatilely and robustly in dynamic and complex environment. Their biological principles are so practical and efficient that inspired we human imitating them in our flying machines. In this paper, we studied a novel bio-inspired collision detector and its application on a quadcopter. The detector is inspired from Lobula giant movement detector (LGMD) neurons in the locusts, and modeled into an STM32F407 Microcontroller Unit (MCU).
    Compared to other collision detecting methods applied on quadcopters, we focused on enhancing the collision accuracy in a bio-inspired way that can considerably increase the computing efficiency during an obstacle detecting task even in complex and dynamic environment. We designed the quadcopter's responding operation to imminent collisions and tested this bio-inspired system in an indoor arena. The observed results from the experiments demonstrated that the LGMD collision detector is feasible to work as a vision module for the quadcopter's collision avoidance task.}
    }