| 1. |
- Cortinhal, Tiago, 1990-
(författare)
-
Semantics-aware Multi-modal Scene Perception for Autonomous Vehicles
- 2024
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Autonomous vehicles represent the pinnacle of modern technological innovation, navigating complex and unpredictable environments. To do so effectively, they rely on a sophisticated array of sensors. This thesis explores two of the most crucial sensors: LiDARs, known for their accuracy in generating detailed 3D maps of the environment, and RGB cameras, essential for processing visual cues critical for navigation. Together, these sensors form a comprehensive perception system that enables autonomous vehicles to operate safely and efficiently.However, the reliability of these vehicles has yet to be tested when key sensors fail. The abrupt failure of a camera, for instance, disrupts the vehicle’s perception system, creating a significant gap in sensory input. This thesis addresses this challenge by introducing a novel multi-modal domain translation framework that integrates LiDAR and RGB camera data while ensuring continuous functionality despite sensor failures. At the core of this framework is an innovative model capable of synthesizing RGB images and their corresponding segment maps from raw LiDAR data by exploiting the scene semantics. The proposed framework stands out as the first of its kind, demonstrating for the first time that the scene semantics can bridge the gap across different domains with distinct data structures, such as unorganized sparse 3D LiDAR point clouds and structured 2D camera data. Thus, this thesis represents a significant leap forward in the field, offering a robust solution to the challenge of RGB data recovery without camera sensors.The practical application of this model is thoroughly explored in the thesis. It involves testing the model’s capability to generate pseudo point clouds from RGB depth estimates, which, when combined with LiDAR data, create an enriched perception dataset. This enriched dataset is pivotal in enhancing object detection capabilities, a fundamental aspect of autonomous vehicle navigation. The quantitative and qualitative evidence reported in this thesis demonstrates that the synthetic generation of data not only compensates for the loss of sensory input but also considerably improves the performance of object detection systems compared to using raw LiDAR data only.By addressing the critical issue of sensor failure and presenting viable solutions, this thesis contributes to enhancing the safety, reliability, and efficiency of autonomous vehicles. It paves the way for further research and developiment, setting a new standard for autonomous vehicle technology in scenarios of sensor malfunctions or adverse environmental conditions.
|
|
| 2. |
- Aein, Mohamad Javad, et al.
(författare)
-
Library of actions : Implementing a generic robot execution framework by using manipulation action semantics
- 2019
-
Ingår i: The international journal of robotics research. - London : Sage Publications. - 0278-3649 .- 1741-3176. ; 38:8, s. 910-934
-
Tidskriftsartikel (refereegranskat)abstract
- Drive-thru-Internet is a scenario in cooperative intelligent transportation systems (C-ITSs), where a road-side unit (RSU) provides multimedia services to vehicles that pass by. Performance of the drive-thru-Internet depends on various factors, including data traffic intensity, vehicle traffic density, and radio-link quality within the coverage area of the RSU, and must be evaluated at the stage of system design in order to fulfill the quality-of-service requirements of the customers in C-ITS. In this paper, we present an analytical framework that models downlink traffic in a drive-thru-Internet scenario by means of a multidimensional Markov process: the packet arrivals in the RSU buffer constitute Poisson processes and the transmission times are exponentially distributed. Taking into account the state space explosion problem associated with multidimensional Markov processes, we use iterative perturbation techniques to calculate the stationary distribution of the Markov chain. Our numerical results reveal that the proposed approach yields accurate estimates of various performance metrics, such as the mean queue content and the mean packet delay for a wide range of workloads. © 2019 IEEE.
|
|
| 3. |
- Ak, Abdullah Cihan, et al.
(författare)
-
Learning Failure Prevention Skills for Safe Robot Manipulation
- 2023
-
Ingår i: IEEE Robotics and Automation Letters. - Piscataway, NJ : IEEE. - 2377-3766. ; 8:12, s. 7994-8001
-
Tidskriftsartikel (refereegranskat)abstract
- Robots are more capable of achieving manipulation tasks for everyday activities than before. However, the safety of manipulation skills that robots employ is still an open problem. Considering all possible failures during skill learning increases the complexity of the process and restrains learning an optimal policy. Nonetheless, safety-focused modularity in the acquisition of skills has not been adequately addressed in previous works. For that purpose, we reformulate skills as base and failure prevention skills, where base skills aim at completing tasks and failure prevention skills aim at reducing the risk of failures to occur. Then, we propose a modular and hierarchical method for safe robot manipulation by augmenting base skills by learning failure prevention skills with reinforcement learning and forming a skill library to address different safety risks. Furthermore, a skill selection policy that considers estimated risks is used for the robot to select the best control policy for safe manipulation. Our experiments show that the proposed method achieves the given goal while ensuring safety by preventing failures. We also show that with the proposed method, skill learning is feasible and our safe manipulation tools can be transferred to the real environment © 2023 IEEE
|
|
| 4. |
- Aksoy, Eren, 1982-, et al.
(författare)
-
SalsaNet : Fast Road and Vehicle Segmentationin LiDAR Point Clouds for Autonomous Driving
- 2020
-
Ingår i: IEEE Intelligent Vehicles Symposium. - Piscataway, N.J. : IEEE. ; , s. 926-932
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we introduce a deep encoder-decoder network, named SalsaNet, for efficient semantic segmentation of 3D LiDAR point clouds. SalsaNet segments the road, i.e. drivable free-space, and vehicles in the scene by employing the Bird-Eye-View (BEV) image projection of the point cloud. To overcome the lack of annotated point cloud data, in particular for the road segments, we introduce an auto-labeling process which transfers automatically generated labels from the camera to LiDAR. We also explore the role of imagelike projection of LiDAR data in semantic segmentation by comparing BEV with spherical-front-view projection and show that SalsaNet is projection-agnostic. We perform quantitative and qualitative evaluations on the KITTI dataset, which demonstrate that the proposed SalsaNet outperforms other state-of-the-art semantic segmentation networks in terms of accuracy and computation time. Our code and data are publicly available at https://gitlab.com/aksoyeren/salsanet.git.
|
|
| 5. |
- Akyol, Gamze, et al.
(författare)
-
A Variational Graph Autoencoder for Manipulation Action Recognition and Prediction
- 2021
-
Konferensbidrag (refereegranskat)abstract
- Despite decades of research, understanding human manipulation activities is, and has always been, one of the most attractive and challenging research topics in computer vision and robotics. Recognition and prediction of observed human manipulation actions have their roots in the applications related to, for instance, human-robot interaction and robot learning from demonstration. The current research trend heavily relies on advanced convolutional neural networks to process the structured Euclidean data, such as RGB camera images. These networks, however, come with immense computational complexity to be able to process high dimensional raw data.Different from the related works, we here introduce a deep graph autoencoder to jointly learn recognition and prediction of manipulation tasks from symbolic scene graphs, instead of relying on the structured Euclidean data. Our network has a variational autoencoder structure with two branches: one for identifying the input graph type and one for predicting the future graphs. The input of the proposed network is a set of semantic graphs which store the spatial relations between subjects and objects in the scene. The network output is a label set representing the detected and predicted class types. We benchmark our new model against different state-of-the-art methods on two different datasets, MANIAC and MSRC-9, and show that our proposed model can achieve better performance. We also release our source code https://github.com/gamzeakyol/GNet.
|
|
| 6. |
- Cooney, Martin, 1980-, et al.
(författare)
-
Exercising with an “Iron Man” : Design for a Robot Exercise Coach for Persons with Dementia
- 2020
-
Ingår i: 29th IEEE International Conference on Robot and Human Interactive Communication. - Piscataway : Institute of Electrical and Electronics Engineers (IEEE). - 9781728160757 - 9781728160764 ; , s. 899-905
-
Konferensbidrag (refereegranskat)abstract
- Socially assistive robots are increasingly being designed to interact with humans in various therapeutical scenarios. We believe that one useful scenario is providing exercise coaching for Persons with Dementia (PWD), which involves unique challenges related to memory and communication. We present a design for a robot that can seek to help a PWD to conduct exercises by recognizing their behaviors and providing appropriate feedback, in an online, multimodal, and engaging way. Additionally, following a mid-fidelity prototyping approach, we report on some observations from an exploratory user study using a Baxter robot; although limited by the sample size and our simplified approach, the results suggested the usefulness of the general scenario, and that the degree to which a robot provides feedback–occasional or continuous– could moderate impressions of attentiveness or fun. Some possibilities for future improvement are outlined, touching on richer recognition and behavior generation strategies based on deep learning and haptic feedback, toward informing next designs. © 2020 IEEE.
|
|
| 7. |
- Cortinhal, Tiago, 1990-, et al.
(författare)
-
Depth- and semantics-aware multi-modal domain translation : Generating 3D panoramic color images from LiDAR point clouds
- 2024
-
Ingår i: Robotics and Autonomous Systems. - Amsterdam : Elsevier. - 0921-8890 .- 1872-793X. ; 171, s. 1-9
-
Tidskriftsartikel (refereegranskat)abstract
- This work presents a new depth-and semantics-aware conditional generative model, named TITAN-Next, for cross-domain image-to-image translation in a multi-modal setup between LiDAR and camera sensors. The proposed model leverages scene semantics as a mid-level representation and is able to translate raw LiDAR point clouds to RGB-D camera images by solely relying on semantic scene segments. We claim that this is the first framework of its kind and it has practical applications in autonomous vehicles such as providing a fail-safe mechanism and augmenting available data in the target image domain. The proposed model is evaluated on the large-scale and challenging Semantic-KITTI dataset, and experimental findings show that it considerably outperforms the original TITAN-Net and other strong baselines by 23.7% margin in terms of IoU. © 2023 The Author(s).
|
|
| 8. |
- Cortinhal, Tiago, 1990-, et al.
(författare)
-
SalsaNext : Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving
- 2021
-
Ingår i: Advances in Visual Computing. - Cham : Springer. - 9783030645595 - 9783030645588 ; , s. 207-222
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we introduce SalsaNext for the uncertainty-aware semantic segmentation of a full 3D LiDAR point cloud in real-time. SalsaNext is the next version of SalsaNet which has an encoder-decoder architecture where the encoder unit has a set of ResNet blocks and the decoder part combines upsampled features from the residual blocks. In contrast to SalsaNet, we introduce a new context module, replace the ResNet encoder blocks with a new residual dilated convolution stack with gradually increasing receptive fields and add the pixel-shuffle layer in the decoder. Additionally, we switch from stride convolution to average pooling and also apply central dropout treatment. To directly optimize the Jaccard index, we further combine the weighted cross entropy loss with Lovász-Softmax loss. We finally inject a Bayesian treatment to compute the epistemic and aleatoric uncertainties for each point in the cloud. We provide a thorough quantitative evaluation on the Semantic-KITTI dataset, which demonstrates that the proposed SalsaNext outperforms other published semantic segmentation networks and achieves 3.6% more accuracy over the previous state-of-the-art method. We also release our source code1. © 2020, Springer Nature Switzerland AG.[1] https://github.com/TiagoCortinhal/SalsaNext
|
|
| 9. |
- Cortinhal, Tiago, 1990-, et al.
(författare)
-
Semantics-aware LiDAR-Only Pseudo Point Cloud Generation for 3D Object Detection
- 2024
-
Konferensbidrag (refereegranskat)abstract
- Although LiDAR sensors are crucial for autonomous systems due to providing precise depth information, they struggle with capturing fine object details, especially at a distance, due to sparse and non-uniform data. Recent advances introduced pseudo-LiDAR, i.e., synthetic dense point clouds, using additional modalities such as cameras to enhance 3D object detection. We present a novel LiDAR-only framework that augments raw scans with denser pseudo point clouds by solely relying on LiDAR sensors and scene semantics, omitting the need for cameras. Our framework first utilizes a segmentation model to extract scene semantics from raw point clouds, and then employs a multi-modal domain translator to generate synthetic image segments and depth cues without real cameras. This yields a dense pseudo point cloud enriched with semantic information. We also introduce a new semantically guided projection method, which enhances detection performance by retaining only relevant pseudo points. We applied our framework to different advanced 3D object detection methods and reported up to 2.9% performance upgrade. We also obtained comparable results on the KITTI 3D object detection dataset, in contrast to other state-of-the-art LiDAR-only detectors.
|
|
| 10. |
- Cortinhal, Tiago, 1990-, et al.
(författare)
-
Semantics-aware Multi-modal Domain Translation : From LiDAR Point Clouds to Panoramic Color Images
- 2021
-
Ingår i: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). - Los Alamitos : IEEE Computer Society. - 9781665401913 - 9781665401920 ; , s. 3032-3041
-
Konferensbidrag (refereegranskat)abstract
- In this work, we present a simple yet effective framework to address the domain translation problem between different sensor modalities with unique data formats. By relying only on the semantics of the scene, our modular generative framework can, for the first time, synthesize a panoramic color image from a given full 3D LiDAR point cloud. The framework starts with semantic segmentation of the point cloud, which is initially projected onto a spherical surface. The same semantic segmentation is applied to the corresponding camera image. Next, our new conditional generative model adversarially learns to translate the predicted LiDAR segment maps to the camera image counterparts. Finally, generated image segments are processed to render the panoramic scene images. We provide a thorough quantitative evaluation on the SemanticKitti dataset and show that our proposed framework outperforms other strong baseline models. Our source code is available at https://github. com/halmstad-University/TITAN-NET. © 2021 IEEE.
|
|
