Song, S, Cheung, N-M, Chandrasekhar, V and Mandal, B (2018) Deep Adaptive Temporal Pooling for Activity Recognition. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries. ACM/IEEE Joint Conference on Digital Libraries. ISSN 1552-5996

[thumbnail of Paper25Jul2018.pdf]
Paper25Jul2018.pdf - Accepted Version

Download (3MB) | Preview


Deep neural networks have recently achieved competitive accuracy for human activity recognition. However, there is room for improvement, especially in modeling of long-term temporal importance and determining the activity relevance of different temporal segments in a video. To address this problem, we propose a learnable and differentiable module: Deep Adaptive Temporal Pooling (DATP). DATP applies a self-attention mechanism to adaptively pool the classification scores of different video segments. Specifically, using frame-level features, DATP regresses importance of different temporal segments, and generates weights for them. Remarkably, DATP is trained using only the video-level label. There is no need of additional supervision except video-level activity class label. We conduct extensive experiments to investigate various input features and different weight models. Experimental results show that DATP can learn to assign large weights to key video segments. More importantly, DATP can improve training of frame-level feature extractor. This is because relevant temporal segments are assigned large weights during back-propagation. Overall, we achieve state-of-the-art performance on UCF101, HMDB51 and Kinetics datasets.

Item Type: Article
Additional Information: © 2018 The Authors (from manuscript). This is the accepted author manuscript (AAM). The final published version (version of record) is available online via ACM at - please refer to any applicable terms of use of the publisher.
Uncontrolled Keywords: Human activity recognition, adaptive temporal pooling
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Faculty of Natural Sciences > School of Computing and Mathematics
Depositing User: Symplectic
Date Deposited: 05 Nov 2018 09:19
Last Modified: 30 Mar 2021 09:28

Actions (login required)

View Item
View Item