| 1. |
|
|
| 2. |
- Yin, Wenjie, et al.
(författare)
-
Controllable Motion Synthesis and Reconstruction with Autoregressive Diffusion Models
- 2023
-
Ingår i: 2023 32ND IEEE INTERNATIONAL CONFERENCE ON ROBOT AND HUMAN INTERACTIVE COMMUNICATION, RO-MAN. - : Institute of Electrical and Electronics Engineers (IEEE). ; , s. 1102-1108
-
Konferensbidrag (refereegranskat)abstract
- Data-driven and controllable human motion synthesis and prediction are active research areas with various applications in interactive media and social robotics. Challenges remain in these fields for generating diverse motions given past observations and dealing with imperfect poses. This paper introduces MoDiff, an autoregressive probabilistic diffusion model over motion sequences conditioned on control contexts of other modalities. Our model integrates a cross-modal Transformer encoder and a Transformer-based decoder, which are found effective in capturing temporal correlations in motion and control modalities. We also introduce a new data dropout method based on the diffusion forward process to provide richer data representations and robust generation. We demonstrate the superior performance of MoDiff in controllable motion synthesis for locomotion with respect to two baselines and show the benefits of diffusion data dropout for robust synthesis and reconstruction of high-fidelity motion close to recorded data.
|
|
| 3. |
- Yin, Wenjie, et al.
(författare)
-
Dance Style Transfer with Cross-modal Transformer
- 2023
-
Ingår i: 2023 IEEE/CVF WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV). - : Institute of Electrical and Electronics Engineers (IEEE). ; , s. 5047-5056
-
Konferensbidrag (refereegranskat)abstract
- We present CycleDance, a dance style transfer system to transform an existing motion clip in one dance style to a motion clip in another dance style while attempting to preserve motion context of the dance. Our method extends an existing CycleGAN architecture for modeling audio sequences and integrates multimodal transformer encoders to account for music context. We adopt sequence length-based curriculum learning to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. We further introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. We perform an extensive ablation study as well as a human study including 30 participants with 5 or more years of dance experience. The results demonstrate that CycleDance generates realistic movements with the target style, significantly outperforming the baseline CycleGAN on naturalness, transfer strength, and content preservation.
|
|
| 4. |
- Yin, Wenjie, et al.
(författare)
-
Graph-based Normalizing Flow for Human Motion Generation and Reconstruction
- 2021
-
Ingår i: 2021 30th IEEE international conference on robot and human interactive communication (RO-MAN). - : Institute of Electrical and Electronics Engineers (IEEE). ; , s. 641-648
-
Konferensbidrag (refereegranskat)abstract
- Data-driven approaches for modeling human skeletal motion have found various applications in interactive media and social robotics. Challenges remain in these fields for generating high-fidelity samples and robustly reconstructing motion from imperfect input data, due to e.g. missed marker detection. In this paper, we propose a probabilistic generative model to synthesize and reconstruct long horizon motion sequences conditioned on past information and control signals, such as the path along which an individual is moving. Our method adapts the existing work MoGlow by introducing a new graph-based model. The model leverages the spatial-temporal graph convolutional network (ST-GCN) to effectively capture the spatial structure and temporal correlation of skeletal motion data at multiple scales. We evaluate the models on a mixture of motion capture datasets of human locomotion with foot-step and bone-length analysis. The results demonstrate the advantages of our model in reconstructing missing markers and achieving comparable results on generating realistic future poses. When the inputs are imperfect, our model shows improvements on robustness of generation.
|
|
| 5. |
- Yin, Wenjie, et al.
(författare)
-
Multimodal dance style transfer
- 2023
-
Ingår i: Machine Vision and Applications. - : Springer Nature. - 0932-8092 .- 1432-1769. ; 34:4
-
Tidskriftsartikel (refereegranskat)abstract
- This paper first presents CycleDance, a novel dance style transfer system that transforms an existing motion clip in one dance style into a motion clip in another dance style while attempting to preserve the motion context of the dance. CycleDance extends existing CycleGAN architectures with multimodal transformer encoders to account for the music context. We adopt a sequence length-based curriculum learning strategy to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. Building upon CycleDance, we further propose StarDance, which enables many-to-many mappings across different styles using a single generator network. Additionally, we introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. To evaluate the performance of our approach, we perform an extensive ablation study and a human study with 30 participants, each with 5 or more years of dance experience. Our experimental results show that our approach can generate realistic movements with the target style, outperforming the baseline CycleGAN and its variants on naturalness, transfer strength, and content preservation. Our proposed approach has potential applications in choreography, gaming, animation, and tool development for artistic and scientific innovations in the field of dance.
|
|
| 6. |
- Yin, Wenjie, et al.
(författare)
-
Scalable Motion Style Transfer with Constrained Diffusion Generation
- 2024
-
Ingår i: Proceedings of the 38th AAAI Conference on Artificial Intelligence. - : Association for the Advancement of Artificial Intelligence (AIAA).
-
Konferensbidrag (refereegranskat)abstract
- Current training of motion style transfer systems relies on consistency losses across style domains to preserve contents, hindering its scalable application to a large number of domains and private data. Recent image transfer works show the potential of independent training on each domain by leveraging implicit bridging between diffusion models, with the content preservation, however, limited to simple data patterns. We address this by imposing biased sampling in backward diffusion while maintaining the domain independence in the training stage. We construct the bias from the source domain keyframes and apply them as the gradient of content constraints, yielding a framework with keyframe manifold constraint gradients (KMCGs). Our validation demonstrates the success of training separate models to transfer between as many as ten dance motion styles. Comprehensive experiments find a significant improvement in preserving motion contents in comparison to baseline and ablative diffusion-based style transfer models. In addition, we perform a human study for a subjective assessment of the quality of generated dance motions. The results validate the competitiveness of KMCGs.
|
|
| 7. |
- Chen, Hongxia, et al.
(författare)
-
PRL2 Phosphatase Promotes Oncogenic KIT Signaling in Leukemia Cells through Modulating CBL Phosphorylation
- 2024
-
Ingår i: Molecular Cancer Research. - 1541-7786. ; 22:1, s. 94-103
-
Tidskriftsartikel (refereegranskat)abstract
- Receptor tyrosine kinase KIT is frequently activated in acute myeloid leukemia (AML). While high PRL2 (PTP4A2) expression is correlated with activation of SCF/KIT signaling in AML, the underlying mechanisms are not fully understood. We discovered that inhibition of PRL2 significantly reduces the burden of oncogenic KIT-driven leukemia and extends leukemic mice survival. PRL2 enhances oncogenic KIT signaling in leukemia cells, promoting their proliferation and survival. We found that PRL2 dephosphorylates CBL at tyrosine 371 and inhibits its activity toward KIT, leading to decreased KIT ubiquitination and enhanced AKT and ERK signaling in leukemia cells.
|
|
| 8. |
- Demir Kanik, Sumeyra Ummuhan, PhD, et al.
(författare)
-
Improving EEG-based Motor Execution Classification for Robot Control
- 2022
-
Ingår i: Proceedings 14th International Conference, SCSM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. - Cham : Springer Nature. ; , s. 65-82
-
Konferensbidrag (refereegranskat)abstract
- Brain Computer Interface (BCI) systems have the potential to provide a communication tool using non-invasive signals which can be applied to various fields including neuro-rehabilitation and entertainment. Interpreting multi-class movement intentions in a real time setting to control external devices such as robotic arms remains to be one of the main challenges in the BCI field. We propose a learning framework to decode upper limb movement intentions before and during the movement execution (ME) with the inclusion of motor imagery (MI) trials. The design of the framework allows the system to evaluate the uncertainty of the classification output and respond accordingly. The EEG signals collected during MI and ME trials are fed into a hybrid architecture consisting of Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) with limited pre-processing. Outcome of the proposed approach shows the potential to anticipate the intended movement direction before the onset of the movement, while waiting to reach a certainty level by potentially observing more EEG data from the beginning of the actual movement before sending control commands to the robot to avoid undesired outcomes. Presented results indicate that both the accuracy and the confidence level of the model improves with the introduction of MI trials right before the movement execution. Our results confirm the possibility of the proposed model to contribute to real-time and continuous decoding of movement directions for robotic applications.
|
|
| 9. |
- Fu, Jia, et al.
(författare)
-
Component atention network for multimodal dance improvisation recognition
- 2023
-
Ingår i: PROCEEDINGS OF THE 25TH INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ICMI 2023. - : Association for Computing Machinery (ACM). ; , s. 114-118
-
Konferensbidrag (refereegranskat)abstract
- Dance improvisation is an active research topic in the arts. Motion analysis of improvised dance can be challenging due to its unique dynamics. Data-driven dance motion analysis, including recognition and generation, is often limited to skeletal data. However, data of other modalities, such as audio, can be recorded and benefit downstream tasks. This paper explores the application and performance of multimodal fusion methods for human motion recognition in the context of dance improvisation. We propose an attention-based model, component attention network (CANet), for multimodal fusion on three levels: 1) feature fusion with CANet, 2) model fusion with CANet and graph convolutional network (GCN), and 3) late fusion with a voting strategy. We conduct thorough experiments to analyze the impact of each modality in different fusion methods and distinguish critical temporal or component features. We show that our proposed model outperforms the two baseline methods, demonstrating its potential for analyzing improvisation in dance.
|
|
| 10. |
- Ghadirzadeh, Ali, et al.
(författare)
-
Human-Centered Collaborative Robots With Deep Reinforcement Learning
- 2021
-
Ingår i: IEEE Robotics and Automation Letters. - : Institute of Electrical and Electronics Engineers (IEEE). - 2377-3766. ; 6:2, s. 566-571
-
Tidskriftsartikel (refereegranskat)abstract
- We present a reinforcement learning based framework for human-centered collaborative systems. The framework is proactive and balances the benefits of timely actions with the risk of taking improper actions by minimizing the total time spent to complete the task. The framework is learned end-to-end in an unsupervised fashion addressing the perception uncertainties and decision making in an integrated manner. The framework is shown to provide more time-efficient coordination between human and robot partners on an example task of packaging compared to alternatives for which perception and decision-making systems are learned independently, using supervised learning. Two important benefits of the proposed approach are that tedious annotation of motion data is avoided, and the learning is performed on-line.
|
|
