Skip to content

PyTorch Code of our WACV2020 paper: Silhouette Guided Point Cloud Reconstruction beyond Occlusion

License

Notifications You must be signed in to change notification settings

zouchuhang/Silhouette-Guided-3D

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

49 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Silhouette-Guided-3D

PyTorch implementation of our WACV 2020 paper: "Silhouette Guided Point Cloud Reconstruction beyond Occlusion"

Our short introduction video

Network architecture:

Requirements

  • Python 3
  • PyTorch >= 0.4.0
  • numpy, scipy, pickle, skimage, sklearn, random, re
  • torchvision
  • Matlab (for FSSR based post process)

Installation

  • Install mve by following the instructions. This is for FSSR based point cloud refinement.
  • Under ./matlab folder, install gptoolbox for Matlab based Poisson-Disc Sampling
  • [Optional] Install Pix3D evaluation toolkit under the current folder. Note that this requires Tensorflow.
  • [Optional] Install PCN evaluation toolkit under the current folder. Note that this requires Tensorflow. PCN toolkit is for object-centered evaluation.
  • [Optional] Install Mask-RCNN benchmark under the current folder. This is for getting visible silhouette for completion in Pix3D (We've included pre-processed results as below).

Download Data and Pre-trained Model

  • Download pre-trained models and put them under the ./model/ folder.
  • Download pre-processed DYCE dataset and put them under the ./data/ folder.
  • Download pre-processed Pix3D dataset and put them under the ./data/ folder. This includes pre-computed complete silhouette and ground truth point clouds rotated w.r.t. camera position. We've excluded examples with incorrect Mask-RCNN detections.
  • Download ShapeNet dataset and put them under the ./data/ folder.
  • Download pre-processed LSUN dataset and put them under the ./data/ folder
  • Download pre-computed result. and put them under the ./result/ folder. This includes point clouds prediction on ShapeNet and Pix3D after FSSR refinement.
  • [Optional] Download ShapeNet rendered images, and put them under ./data/ShapeNet/ folder. This is for comparing to object-centered point cloud reconstruction approach.
  • [Optional] Download ShapeNet object-centered point cloud ground truth, , and put them under ./data/ShapeNet/ folder. This is for comparing to object-centered point cloud reconstruction approach.

Training

  • Point cloud reconstruction

    python train.py
    python test.py
    

    This will save network predictions for the downstream FSSR post-refinement step.

  • Silhouette completion First train on DYCE dataset:

    python train_sc.py
    

    Then finetune on Pix3D dataset, using 5-fold cross validation ( you will need to run it 5 times by changing the fold number in L32-35 ):

    python train_sc_ft.py
    python test_sc_pix3d.py
    
  • Silhouette guidede point cloud reconstruction

    python train_occ.py
    python test_rec_pix3d.py
    

    Then perform FSSR post-refinement step as describe below

Surface Based Point Clouds Refinement

  • Start matlab
    cd matlab
    ./matlab
    
  • pre-compute FSSR params (per-pixel normal and scale), change folder name based on your saved network predictions path
    FssrPostRefine
    
  • FSSR Here we provide the sample batch-process code (need to go back to the main folder):
    cd ..
    python fssr_batch_process.py
    
  • smoothing
    cd matlab
    preComputeFssrParam
    cd ..
    
    This produces the refined point clouds for evaluation.

Evaluation

  • Pix3D
    • ICP-based fitting since Pix3D ground truth is object-centered (you can skip this step since we've included pre-computered ground truth and predictions). Code is derived from 3D-LMNET, which also includes the ground truth point cloud.
      cd pcn
      python metrics_pix3d.py
      cd ..
      
    • You need to use TensorFlow 3.0+ to run the evaluation:
      cd pix3d/eval/
      python eval_pix3d.py
      cd ../../
      
  • ShapeNet
    • You need to use TensorFlow 3.0+ to run the evaluation:
      cd pcn
      python eval_shapenet.py
      cd ..
      
    • To compare with object-centered point cloud prediction appraoch (3D-LMNET), you need to perform ICP-based fitting first:
      cd pcn
      python metrics_shapenet.py
      cd ../pix3d/eval/
      python eval_shapenet_object_centered.py
      cd ../../
      
  • DYCE (silhouette completion)
    • This is PyTorch based
      python test_sc_DYCE.py
      

Citation

Please cite our paper for any purpose of usage.

@inproceedings{zou2020silhouette,
  title={Silhouette Guided Point Cloud Reconstruction beyond Occlusion},
  author={Zou, Chuhang and Hoiem, Derek},
  booktitle={The IEEE Winter Conference on Applications of Computer Vision},
  pages={41--50},
  year={2020}
}

About

PyTorch Code of our WACV2020 paper: Silhouette Guided Point Cloud Reconstruction beyond Occlusion

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published