webpage | pdf | abstract | bibtex |

The aim of this work is to develop an approach that enables Unmanned Aerial System (UAS) to efficiently learn to navigate in large-scale urban environments and transfer their acquired expertise to novel environments. To achieve this, we propose a meta curriculum training scheme. First, the meta training allows the agent to learn a master policy to generalize across tasks. The resulting model is then fine-tuned on the downstream tasks. We organize the training curriculum in a hierarchical manner, such that the agent is guided from coarse to fine towards the target task. In addition, we introduce Incremental Self-Adaptive Reinforcement learning (ISAR), an algorithm that combines the ideas of incremental learning and meta reinforcement learning (MRL). In contrast to traditional reinforcement learning (RL), which focuses on acquiring a policy for a specific task, MRL aims to learn a policy with fast transfer ability to novel tasks. However, the training process of MRL is time-consuming, whereas our proposed ISAR algorithm achieves faster convergence than conventional MRL algorithm. We evaluate the proposed methodologies in simulated environments and demonstrate that using such training philosophy in conjunction with the ISAR algorithm significantly improves the convergence speed for navigation in large-scale cities and the adaptation proficiency in novel environments.

  @Article{isprs-annals-X-2-2024-105-2024,
    AUTHOR = {Han, Y. and Toth, C. and Yilmaz, A.},
    TITLE = {UAS Visual Navigation in Large and Unseen Environments via a Meta Agent},
    JOURNAL = {ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
    VOLUME = {X-2-2024},
    YEAR = {2024},
    PAGES = {105--112},
    URL = {https://isprs-annals.copernicus.org/articles/X-2-2024/105/2024/},
    DOI = {10.5194/isprs-annals-X-2-2024-105-2024}
    }

webpage | pdf | abstract | bibtex |

We propose Sparse Imitation Reinforcement Learning (SIRL), a hybrid end-to-end control policy that combines the sparse expert driving knowledge with reinforcement learning (RL) policy for autonomous driving (AD) task in CARLA simulation environment. The sparse expert is designed based on hand-crafted rules which is suboptimal but provides a risk-averse strategy by enforcing experience for critical scenarios such as pedestrian and vehicle avoidance, and traffic light detection. As it has been demonstrated, training a RL agent from scratch is data-inefficient and time consuming particularly for the urban driving task, due to the complexity of situations stemming from the vast size of state space. Our SIRL strategy provides a solution to solve these problems by fusing the output distribution of the sparse expert policy and the RL policy to generate a composite driving policy. With the guidance of the sparse expert during the early training stage, SIRL strategy accelerates the training process and keeps the RL exploration from causing a catastrophe outcome, and ensures safe exploration. To some extent, the SIRL agent is imitating the driving expert’s behavior. At the same time, it continuously gains knowledge during training therefore it keeps making improvement beyond the sparse expert, and can surpass both the sparse expert and a traditional RL agent. We experimentally validate the efficacy of proposed SIRL approach in a complex urban scenario within the CARLA simulator. Besides, we compare the SIRL agent’s performance for risk-averse exploration and high learning efficiency with the traditional RL approach. We additionally demonstrate the SIRL agent’s generalization ability to transfer the driving skill to unseen environment.

  @INPROCEEDINGS {9956121,
    author = { Han, Yuci and Yilmaz, Alper },
    booktitle = { 2022 26th International Conference on Pattern Recognition (ICPR) },
    title = {{ Learning to Drive Using Sparse Imitation Reinforcement Learning }},
    year = {2022},
  }

pdf | abstract | bibtex |

This paper presents a novel end-to-end Unmanned Aerial System (UAS) navigation approach for long-range visual navigation in the real world. Inspired by dual-process visual navigation system of human's instinct: environment understanding and landmark recognition, we formulate the UAS navigation task into two same phases. Our system combines the reinforce-ment learning (RL) and image matching approaches. First, the agent learns the navigation policy using RL in the specified environment. To achieve this, we design an interactive UASNAV environment for the training process. Once the agent learns the navigation policy, which means ‘familiarized themselves with the environment’, we let the UAS fly in the real world to recognize the landmarks using image matching method and take action according to the learned policy. During the navigation process, the UAS is embedded with single camera as the only visual sensor. We demonstrate that the UAS can learn navigating to the destination hundreds meters away from the starting point with the shortest path in the real world scenario.

  @INPROCEEDINGS{9967103,
    author={Han, Yuci and Wei, Jianli and Yilmaz, Alper},
    booktitle={2022 IEEE Sensors}, 
    title={UAS Navigation in the Real World Using Visual Observation}, 
    year={2022},
    volume={},
    number={},
    pages={1-4},
    keywords={Meters;Training;Visualization;Image recognition;Satellites;Navigation;Image matching;deep reinforcement learning;image matching;visual navigation},
    doi={10.1109/SENSORS52175.2022.9967103}}

pdf | abstract | bibtex |

In this work, we propose an approach for an autonomous agent that learns to navigate in an unknown map in a real-world environment. Recognizing that the real-world environment is changing overtime such as road-closure happening due to construction work, a key contribution of our paper is adopt the dynamic adaptation characteristic of the reinforcement learning approach and develop a dynamic routing ability for our agent. Our method is based on the Q-learning algorithm and modifies it into a double-critic Qlearning model (DCQN) that only uses visual input without other aids such as GPS. Our treatment of the problem enables the agent to learn the navigation policy while interacting with the environment. We demonstrate that the agent can learn navigating to the destination kilometers away from the starting point in a real world scenario and has the ability to respond to environment changes while learning to adjust the routing plan dynamically by adjusting the old knowledge.

  @Article{isprs-annals-V-1-2021-145-2021,
    AUTHOR = {Han, Y. and Yilmaz, A.},
    TITLE = {DYNAMIC ROUTING FOR NAVIGATION IN CHANGING UNKNOWN MAPS USING DEEP REINFORCEMENT LEARNING},
    JOURNAL = {ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
    VOLUME = {V-1-2021},
    YEAR = {2021},
    }
  

abstract

• Developed a neuro-symbolic approach to control UAS with natural language instructions.
• Leveraged the in-context learning ability of large language models (LLM) to generate Python-like modular programs, which are then executed to control the drone. Each line of the generated program may invoke one of several off-the-shelf predefined modules.
• Deployed and tested in an Unreal Engine simulation environment with the AirSim plugin..

abstract

• Deployed an iRobot Create 2 platform with a single Orbbec camera and NVIDIA Jetson Nano to enable autonomous navigation in real indoor environments.
• Enabled real-time environment interaction and precise point-goal navigation (e.g., back door in the video).