.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "mmwave_PC/mmwave_PC_hpe_tutorial.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_mmwave_PC_mmwave_PC_hpe_tutorial.py: Tutorial for Human Pose Estimation ============================================================== .. GENERATED FROM PYTHON SOURCE LINES 8-9 In[1]: .. GENERATED FROM PYTHON SOURCE LINES 9-16 .. code-block:: Python import yaml import torch import torch.nn as nn from tqdm import tqdm import os .. GENERATED FROM PYTHON SOURCE LINES 17-19 Dataset with MetaFi: ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 21-24 Point cloud Pose reconstruction dataset collected by Ti 6843 mmWave radar. 40 subjects are included and the human poses are obtained by 2 RGB camera. We provide cross-subject experiment settings with all daily activities. In the library, we provide a dataloader to use mmWave PC data, and predict these human poses. .. GENERATED FROM PYTHON SOURCE LINES 27-29 Load the data ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 31-32 In[3]: .. GENERATED FROM PYTHON SOURCE LINES 32-39 .. code-block:: Python from pysensing.mmwave.PC.dataset.hpe import load_hpe_dataset # The path contains the radHAR dataset train_dataset, test_dataset = load_hpe_dataset("MetaFi") .. rst-class:: sphx-glr-script-out .. code-block:: none Try to download MetaFi dateset in /home/kemove/yyz/av-gihub/tutorials/mmwave_PC_source/mmfi Downloading MetaFi to /home/kemove/yyz/av-gihub/tutorials/mmwave_PC_source/mmfi.zip... Downloading: 0%| | 0.00/260M [00:00 .. GENERATED FROM PYTHON SOURCE LINES 55-57 Create model ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 59-62 mmFi utilizes PointTransformer model as a baseline hpe method. From model.hpe, we can import desired hpe model designed for mmWave PC. The model parameter for PointTransformer reimplemented for mmFi is as follows: .. GENERATED FROM PYTHON SOURCE LINES 64-65 In[7]: .. GENERATED FROM PYTHON SOURCE LINES 65-75 .. code-block:: Python from pysensing.mmwave.PC.model.hpe import PointTransformerReg model = PointTransformerReg( input_dim = 5, nblocks = 5, n_p = 17 ) print(model) .. rst-class:: sphx-glr-script-out .. code-block:: none PointTransformerReg( (backbone): Backbone( (fc1): Sequential( (0): Linear(in_features=5, out_features=32, bias=True) (1): ReLU() (2): Linear(in_features=32, out_features=32, bias=True) ) (transformer1): TransformerBlock( (fc1): Linear(in_features=32, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=32, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (transition_downs): ModuleList( (0): TransitionDown( (conv1): Sequential( (0): Conv1d(35, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(64, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (1): TransitionDown( (conv1): Sequential( (0): Conv1d(67, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(128, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (2): TransitionDown( (conv1): Sequential( (0): Conv1d(131, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (3): TransitionDown( (conv1): Sequential( (0): Conv1d(259, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (transformers): ModuleList( (0): TransformerBlock( (fc1): Linear(in_features=64, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=64, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (1): TransformerBlock( (fc1): Linear(in_features=128, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=128, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (2): TransformerBlock( (fc1): Linear(in_features=256, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=256, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (3): TransformerBlock( (fc1): Linear(in_features=512, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=512, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) ) ) (transformer): Transformer( (layers): ModuleList( (0-4): 5 x ModuleList( (0): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): Attention( (to_k): Linear(in_features=512, out_features=512, bias=False) (to_v): Linear(in_features=512, out_features=512, bias=False) (to_q): Linear(in_features=512, out_features=512, bias=False) (to_out): Sequential( (0): Linear(in_features=512, out_features=512, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) ) ) ) ) (1): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): FeedForward( (net): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=256, out_features=512, bias=True) (4): Dropout(p=0.0, inplace=False) ) ) ) ) ) ) ) (fc2): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): Dropout(p=0.0, inplace=False) (2): ReLU() (3): Linear(in_features=256, out_features=32, bias=True) ) (fc3): Sequential( (0): ReLU() (1): Linear(in_features=32, out_features=64, bias=True) (2): Dropout(p=0.0, inplace=False) (3): ReLU() (4): Linear(in_features=64, out_features=3, bias=True) ) ) .. GENERATED FROM PYTHON SOURCE LINES 76-77 A shortcut for loading the hpe model to avoid the tedious hyper-parameter setting. .. GENERATED FROM PYTHON SOURCE LINES 80-81 In[8]: .. GENERATED FROM PYTHON SOURCE LINES 81-88 .. code-block:: Python from pysensing.mmwave.PC.model.hpe import load_hpe_model model = load_hpe_model("MetaFi", "PointTransformer") print(model) .. rst-class:: sphx-glr-script-out .. code-block:: none PointTransformerReg( (backbone): Backbone( (fc1): Sequential( (0): Linear(in_features=5, out_features=32, bias=True) (1): ReLU() (2): Linear(in_features=32, out_features=32, bias=True) ) (transformer1): TransformerBlock( (fc1): Linear(in_features=32, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=32, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (transition_downs): ModuleList( (0): TransitionDown( (conv1): Sequential( (0): Conv1d(35, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(64, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (1): TransitionDown( (conv1): Sequential( (0): Conv1d(67, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(128, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (2): TransitionDown( (conv1): Sequential( (0): Conv1d(131, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (3): TransitionDown( (conv1): Sequential( (0): Conv1d(259, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (transformers): ModuleList( (0): TransformerBlock( (fc1): Linear(in_features=64, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=64, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (1): TransformerBlock( (fc1): Linear(in_features=128, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=128, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (2): TransformerBlock( (fc1): Linear(in_features=256, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=256, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (3): TransformerBlock( (fc1): Linear(in_features=512, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=512, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) ) ) (transformer): Transformer( (layers): ModuleList( (0-4): 5 x ModuleList( (0): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): Attention( (to_k): Linear(in_features=512, out_features=512, bias=False) (to_v): Linear(in_features=512, out_features=512, bias=False) (to_q): Linear(in_features=512, out_features=512, bias=False) (to_out): Sequential( (0): Linear(in_features=512, out_features=512, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) ) ) ) ) (1): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): FeedForward( (net): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=256, out_features=512, bias=True) (4): Dropout(p=0.0, inplace=False) ) ) ) ) ) ) ) (fc2): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): Dropout(p=0.0, inplace=False) (2): ReLU() (3): Linear(in_features=256, out_features=32, bias=True) ) (fc3): Sequential( (0): ReLU() (1): Linear(in_features=32, out_features=64, bias=True) (2): Dropout(p=0.0, inplace=False) (3): ReLU() (4): Linear(in_features=64, out_features=3, bias=True) ) ) .. GENERATED FROM PYTHON SOURCE LINES 89-91 Model Train ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 93-96 pysensing library support quick training of model with the following steps. The training interface incorporates pytorch loss functions, optimizers and dataloaders to facilate training. An example is provided for how to define the aforemetioned terms. .. GENERATED FROM PYTHON SOURCE LINES 99-100 In[11]: .. GENERATED FROM PYTHON SOURCE LINES 100-116 .. code-block:: Python # Create pytorch dataloaders train_loader = torch.utils.data.DataLoader(train_dataset, shuffle=True, batch_size=16, num_workers=16) test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=16, shuffle=False, num_workers=16) # Define pytorch loss function as criterion criterion = nn.CrossEntropyLoss() # Define pytorch optimizer for training optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) # GPU acceleration with cuda device = torch.device("cuda" if torch.cuda.is_available() else "cpu") .. GENERATED FROM PYTHON SOURCE LINES 117-118 A quick training using hpe_train. The resulted model parameters will be saved into "train_{num_epochs}.pth". .. GENERATED FROM PYTHON SOURCE LINES 120-121 In[12]: .. GENERATED FROM PYTHON SOURCE LINES 121-128 .. code-block:: Python # Pysensing training interface from pysensing.mmwave.PC.inference.hpe import hpe_train # hpe_train(model, train_loader, num_epochs=1, optimizer=optimizer, criterion=criterion, device=device) .. GENERATED FROM PYTHON SOURCE LINES 129-131 Model inference ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 133-135 Load the pretrained model, e.g. from https://pysensing.oss-ap-southeast-1.aliyuncs.com/pretrain/mmwave_pc/hpe/MetaFi_PointTransformer.pth , and perform human pose estimation! .. GENERATED FROM PYTHON SOURCE LINES 137-138 In[13]: .. GENERATED FROM PYTHON SOURCE LINES 138-145 .. code-block:: Python # load pretrained model from pysensing.mmwave.PC.inference import load_pretrain model = load_pretrain(model, "MetaFi", "PointTransformer").to(device) model.eval() .. rst-class:: sphx-glr-script-out .. code-block:: none Use pretrained model! PointTransformerReg( (backbone): Backbone( (fc1): Sequential( (0): Linear(in_features=5, out_features=32, bias=True) (1): ReLU() (2): Linear(in_features=32, out_features=32, bias=True) ) (transformer1): TransformerBlock( (fc1): Linear(in_features=32, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=32, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (transition_downs): ModuleList( (0): TransitionDown( (conv1): Sequential( (0): Conv1d(35, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(64, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (1): TransitionDown( (conv1): Sequential( (0): Conv1d(67, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(128, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (2): TransitionDown( (conv1): Sequential( (0): Conv1d(131, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (3): TransitionDown( (conv1): Sequential( (0): Conv1d(259, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (transformers): ModuleList( (0): TransformerBlock( (fc1): Linear(in_features=64, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=64, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (1): TransformerBlock( (fc1): Linear(in_features=128, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=128, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (2): TransformerBlock( (fc1): Linear(in_features=256, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=256, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (3): TransformerBlock( (fc1): Linear(in_features=512, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=512, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) ) ) (transformer): Transformer( (layers): ModuleList( (0-4): 5 x ModuleList( (0): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): Attention( (to_k): Linear(in_features=512, out_features=512, bias=False) (to_v): Linear(in_features=512, out_features=512, bias=False) (to_q): Linear(in_features=512, out_features=512, bias=False) (to_out): Sequential( (0): Linear(in_features=512, out_features=512, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) ) ) ) ) (1): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): FeedForward( (net): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=256, out_features=512, bias=True) (4): Dropout(p=0.0, inplace=False) ) ) ) ) ) ) ) (fc2): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): Dropout(p=0.0, inplace=False) (2): ReLU() (3): Linear(in_features=256, out_features=32, bias=True) ) (fc3): Sequential( (0): ReLU() (1): Linear(in_features=32, out_features=64, bias=True) (2): Dropout(p=0.0, inplace=False) (3): ReLU() (4): Linear(in_features=64, out_features=3, bias=True) ) ) .. GENERATED FROM PYTHON SOURCE LINES 146-147 Test the model on testing dataset. .. GENERATED FROM PYTHON SOURCE LINES 149-150 In[14]: .. GENERATED FROM PYTHON SOURCE LINES 150-153 .. code-block:: Python from pysensing.mmwave.PC.inference.hpe import hpe_test # hpe_test(model, test_loader, criterion=criterion, device=device) .. GENERATED FROM PYTHON SOURCE LINES 154-155 Model inference on sample and deep feature embedding of input modality in HPE task. .. GENERATED FROM PYTHON SOURCE LINES 157-158 In[15]: .. GENERATED FROM PYTHON SOURCE LINES 158-173 .. code-block:: Python # Model inference idx = 5 points, pose= test_dataset.__getitem__(idx) points = torch.tensor(points).unsqueeze(0).float().to(device) predicted_result = model(points) print("The predicted pose is {}, while the ground truth is {}".format(predicted_result.cpu(),pose)) # Deep feature embedding from pysensing.mmwave.PC.inference.embedding import embedding emb = embedding(input = points, model=model, dataset_name = "MetaFi", model_name = "PointTransformer", device=device) print("The shape of feature embedding is: ", emb.shape) .. rst-class:: sphx-glr-script-out .. code-block:: none The predicted pose is tensor([[[-0.1173, 0.0153, 3.0501], [-0.2115, 0.0202, 3.0561], [-0.2253, 0.3597, 3.0622], [-0.2422, 0.7359, 3.0763], [ 0.0275, -0.0367, 3.0689], [ 0.0280, 0.3391, 3.0851], [ 0.0785, 0.7613, 3.1083], [-0.1009, -0.2694, 3.0435], [-0.0910, -0.5694, 3.0297], [-0.0988, -0.6771, 3.0037], [-0.1014, -0.7289, 3.0262], [ 0.0349, -0.5211, 3.0320], [ 0.2234, -0.4633, 2.9764], [ 0.1241, -0.5345, 2.8527], [-0.1986, -0.5264, 3.0985], [-0.3518, -0.4575, 3.0666], [-0.2247, -0.4778, 2.9471]]], grad_fn=), while the ground truth is tensor([[-0.0626, -0.0378, 3.3111], [-0.1724, -0.0395, 3.3111], [-0.1786, 0.3689, 3.3083], [-0.2026, 0.7605, 3.3111], [ 0.0473, -0.0362, 3.3111], [ 0.0633, 0.3689, 3.3111], [ 0.0673, 0.7605, 3.3111], [-0.0685, -0.3322, 3.3016], [-0.0744, -0.6267, 3.2920], [-0.0642, -0.7458, 3.2512], [-0.0653, -0.8049, 3.2850], [ 0.0914, -0.5671, 3.3118], [ 0.3367, -0.5248, 3.3104], [ 0.2930, -0.5677, 3.0678], [-0.2505, -0.5671, 3.3131], [-0.5012, -0.5671, 3.3116], [-0.4514, -0.5674, 3.0686]]) The shape of feature embedding is: torch.Size([1, 17, 32]) .. GENERATED FROM PYTHON SOURCE LINES 174-176 mmDiff: diffusion model for mmWave radar HPE ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 178-185 Load Diffusion Runner with model initialized. This process will define the setting for model and dataset. Currently two settings are implemented: 1. "mmBody + P4Transformer": Phase 1: Input [b, 4, 5000, 6]; Output: [b, 17, 3] and [b, 17, 64]. Phase 2: GRC, LRC, TMC, SLC 2. "MetaFi + PointTransformer": Phase 1: Input [b, 5, 150, 5]; Output: [b, 17, 3] and [b, 17, 32]. Phase 2: GRC, TMC, SLC .. GENERATED FROM PYTHON SOURCE LINES 187-188 In[16]: .. GENERATED FROM PYTHON SOURCE LINES 188-192 .. code-block:: Python from pysensing.mmwave.PC.model.hpe.mmDiff.load_mmDiff import load_mmDiff mmDiffRunner = load_mmDiff("MetaFi") .. rst-class:: sphx-glr-script-out .. code-block:: none Self.model_feat vadility passes. MMdiff using PointTransformer as feature extractor. .. GENERATED FROM PYTHON SOURCE LINES 193-196 Phase 1 Training: Can train phase 1 from scratch (is_train = True) or load pretrained phase 1 model (is_train = False). Set is_save = True to facilitate phase 2 training acceleration. If phase 1 features are saved, set is_save = False. .. GENERATED FROM PYTHON SOURCE LINES 198-199 In[17]: .. GENERATED FROM PYTHON SOURCE LINES 199-202 .. code-block:: Python mmDiffRunner.phase1_train(train_dataset, test_dataset, is_train=False, is_save=False) .. rst-class:: sphx-glr-script-out .. code-block:: none Phase 1 use pretrained model! .. GENERATED FROM PYTHON SOURCE LINES 203-204 Phase 1 can also receive self defined model and the model should follow the setting defined above. The Self-defined model should output coarse joints and coarse joint features. .. GENERATED FROM PYTHON SOURCE LINES 206-207 In[18]: .. GENERATED FROM PYTHON SOURCE LINES 207-218 .. code-block:: Python # Self defined model should output coarse joints and coarse joint features from pysensing.mmwave.PC.model.hpe.pointTrans import PointTransformerReg_feat model = PointTransformerReg_feat( input_dim = 5, nblocks = 5, n_p = 17 ) print(model) mmDiffRunner.phase1_train(train_dataset, test_dataset, model_self=model, is_train=False, is_save=False) .. rst-class:: sphx-glr-script-out .. code-block:: none PointTransformerReg_feat( (backbone): Backbone( (fc1): Sequential( (0): Linear(in_features=5, out_features=32, bias=True) (1): ReLU() (2): Linear(in_features=32, out_features=32, bias=True) ) (transformer1): TransformerBlock( (fc1): Linear(in_features=32, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=32, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (transition_downs): ModuleList( (0): TransitionDown( (conv1): Sequential( (0): Conv1d(35, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(64, 64, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (1): TransitionDown( (conv1): Sequential( (0): Conv1d(67, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(128, 128, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (2): TransitionDown( (conv1): Sequential( (0): Conv1d(131, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (3): TransitionDown( (conv1): Sequential( (0): Conv1d(259, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) (conv2): Sequential( (0): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (transformers): ModuleList( (0): TransformerBlock( (fc1): Linear(in_features=64, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=64, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (1): TransformerBlock( (fc1): Linear(in_features=128, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=128, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (2): TransformerBlock( (fc1): Linear(in_features=256, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=256, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) (3): TransformerBlock( (fc1): Linear(in_features=512, out_features=128, bias=True) (fc2): Linear(in_features=128, out_features=512, bias=True) (fc_delta): Sequential( (0): Linear(in_features=3, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (fc_gamma): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=128, bias=True) ) (w_qs): Linear(in_features=128, out_features=128, bias=False) (w_ks): Linear(in_features=128, out_features=128, bias=False) (w_vs): Linear(in_features=128, out_features=128, bias=False) ) ) ) (transformer): Transformer( (layers): ModuleList( (0-4): 5 x ModuleList( (0): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): Attention( (to_k): Linear(in_features=512, out_features=512, bias=False) (to_v): Linear(in_features=512, out_features=512, bias=False) (to_q): Linear(in_features=512, out_features=512, bias=False) (to_out): Sequential( (0): Linear(in_features=512, out_features=512, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) ) ) ) ) (1): Residual( (fn): PreNorm( (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (fn): FeedForward( (net): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): GELU(approximate='none') (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=256, out_features=512, bias=True) (4): Dropout(p=0.0, inplace=False) ) ) ) ) ) ) ) (fc2): Sequential( (0): Linear(in_features=512, out_features=256, bias=True) (1): Dropout(p=0.0, inplace=False) (2): ReLU() (3): Linear(in_features=256, out_features=32, bias=True) ) (fc3): Sequential( (0): ReLU() (1): Linear(in_features=32, out_features=64, bias=True) (2): Dropout(p=0.0, inplace=False) (3): ReLU() (4): Linear(in_features=64, out_features=3, bias=True) ) ) Self.model_feat vadility passes. Phase 1 use self defined model! .. GENERATED FROM PYTHON SOURCE LINES 219-220 Phase 2 Training: Can train from scratch (is_train = True) or load pretrained phase 2 model (is_train = False). .. GENERATED FROM PYTHON SOURCE LINES 222-223 In[19]: .. GENERATED FROM PYTHON SOURCE LINES 223-226 .. code-block:: Python mmDiffRunner.phase2_train(train_loader = None, is_train = False) .. rst-class:: sphx-glr-script-out .. code-block:: none Phase 2 use pretrained model! .. GENERATED FROM PYTHON SOURCE LINES 227-228 Testing mmDiff .. GENERATED FROM PYTHON SOURCE LINES 230-231 In[20]: .. GENERATED FROM PYTHON SOURCE LINES 231-233 .. code-block:: Python #mmDiffRunner.test() .. rst-class:: sphx-glr-timing **Total running time of the script:** (4 minutes 45.885 seconds) .. _sphx_glr_download_mmwave_PC_mmwave_PC_hpe_tutorial.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: mmwave_PC_hpe_tutorial.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: mmwave_PC_hpe_tutorial.py ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_