Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenes

We consider the problem of recovering a single person's 3D human mesh from in-the-wild crowded scenes. While much progress has been in 3D human mesh estimation, existing methods struggle when test input has crowded scenes. The first reason for the failure is a domain gap between training and testing data. A motion capture dataset, which provides accurate 3D labels for training, lacks crowd data and impedes a network from learning crowded scene-robust image features of a target person. The second reason is a feature processing that spatially averages the feature map of a localized bounding box containing multiple people. Averaging the whole feature map makes a target person's feature indistinguishable from others. We present 3DCrowdNet that firstly explicitly targets in-the-wild crowded scenes and estimates a robust 3D human mesh by addressing the above issues. First, we leverage 2D human pose estimation that does not require a motion capture dataset with 3D labels for training and does not suffer from the domain gap. Second, we propose a joint-based regressor that distinguishes a target person's feature from others. Our joint-based regressor preserves the spatial activation of a target by sampling features from the target's joint locations and regresses human model parameters. As a result, 3DCrowdNet learns target-focused features and effectively excludes the irrelevant features of nearby persons. We conduct experiments on various benchmarks and prove the robustness of 3DCrowdNet to the in-the-wild crowded scenes both quantitatively and qualitatively. The code is available at https://github.com/hongsukchoi/3DCrowdNet_RELEASE.Comment: Accepted to CVPR 2022, 16 pages including the supplementary materia

Three Recipes for Better 3D Pseudo-GTs of 3D Human Mesh Estimation in the Wild

Recovering 3D human mesh in the wild is greatly challenging as in-the-wild (ITW) datasets provide only 2D pose ground truths (GTs). Recently, 3D pseudo-GTs have been widely used to train 3D human mesh estimation networks as the 3D pseudo-GTs enable 3D mesh supervision when training the networks on ITW datasets. However, despite the great potential of the 3D pseudo-GTs, there has been no extensive analysis that investigates which factors are important to make more beneficial 3D pseudo-GTs. In this paper, we provide three recipes to obtain highly beneficial 3D pseudo-GTs of ITW datasets. The main challenge is that only 2D-based weak supervision is allowed when obtaining the 3D pseudo-GTs. Each of our three recipes addresses the challenge in each aspect: depth ambiguity, sub-optimality of weak supervision, and implausible articulation. Experimental results show that simply re-training state-of-the-art networks with our new 3D pseudo-GTs elevates their performance to the next level without bells and whistles. The 3D pseudo-GT is publicly available in https://github.com/mks0601/NeuralAnnot_RELEASE.Comment: Published at CVPRW 202

HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image

This paper presents a method to learn hand-object interaction prior for reconstructing a 3D hand-object scene from a single RGB image. The inference as well as training-data generation for 3D hand-object scene reconstruction is challenging due to the depth ambiguity of a single image and occlusions by the hand and object. We turn this challenge into an opportunity by utilizing the hand shape to constrain the possible relative configuration of the hand and object geometry. We design a generalizable implicit function, HandNeRF, that explicitly encodes the correlation of the 3D hand shape features and 2D object features to predict the hand and object scene geometry. With experiments on real-world datasets, we show that HandNeRF is able to reconstruct hand-object scenes of novel grasp configurations more accurately than comparable methods. Moreover, we demonstrate that object reconstruction from HandNeRF ensures more accurate execution of a downstream task, such as grasping for robotic hand-over.Comment: 9 pages, 4 tables, 7 figure

New solvation free energy function comprising intermolecular solvation and intramolecular self-solvation terms

Abstract Solvation free energy is a fundamental thermodynamic quantity that should be determined to estimate various physicochemical properties of a molecule and the desolvation cost for its binding to macromolecular receptors. Here, we propose a new solvation free energy function through the improvement of the solvent-contact model, and test its applicability in estimating the solvation free energies of organic molecules with varying sizes and shapes. This new solvation free energy function is constructed by combining the existing solute-solvent interaction term with the self-solvation term that reflects the effects of intramolecular interactions on solvation. Four kinds of atomic parameters should be determined in this solvation model: atomic fragmental volume, maximum atomic occupancy, atomic solvation, and atomic self-solvation parameters. All of these parameters for total 37 atom types are optimized by the operation of a standard genetic algorithm in such a way to minimize the difference between the experimental solvation free energies and those calculated by the solvation free energy function for 362 organic molecules. The solvation free energies estimated from the new solvation model compare well with the experimental results with the associated squared correlation coefficients of 0.88 and 0.85 for training and test sets, respectively. The present solvation model is thus expected to be useful for estimating the solvation free energies of organic molecules.</p

Computational Prediction of Molecular Hydration Entropy with Hybrid Scaled Particle Theory and Free-Energy Perturbation Method

Despite the importance of the knowledge of molecular hydration entropy (Δ<i>S</i>_hyd) in chemical and biological processes, the exact calculation of Δ<i><i>S</i></i>_hyd is very difficult, because of the complexity in solute–water interactions. Although free-energy perturbation (FEP) methods have been employed quite widely in the literature, the poor convergent behavior of the van der Waals interaction term in the potential function limited the accuracy and robustness. In this study, we propose a new method for estimating Δ<i><i>S</i></i>_hyd by means of combining the FEP approach and the scaled particle theory (or information theory) to separately calculate the electrostatic solute–water interaction term (Δ<i><i>S</i></i>_elec) and the hydrophobic contribution approximated by the cavity formation entropy (Δ<i><i>S</i></i>_cav), respectively. Decomposition of Δ<i><i>S</i></i>_hyd into Δ<i><i>S</i></i>_cav and Δ<i><i>S</i></i>_elec terms is found to be very effective with a substantial accuracy enhancement in Δ<i><i>S</i></i>_hyd estimation, when compared to the conventional full FEP calculations. Δ<i><i>S</i></i>_cav appears to dominate over Δ<i><i>S</i></i>_elec in magnitude, even in the case of polar solutes, implying that the major contribution to the entropic cost for hydration comes from the formation of a solvent-excluded volume. Our hybrid scaled particle theory and FEP method is thus found to enhance the accuracy of Δ<i><i>S</i></i>_hyd prediction by effectively complementing the conventional full FEP method

Enhanced open circuit voltage by hydrophilic ionic liquids as buffer layer in conjugated polymer-nanoporous titania hybrid solar cells

We demonstrate the fabrication of a nanoporous titania (NP-TiO(2)) network structure by using a polystylene-block-poly(4-vinylpyridine) (PS-b-P4VP) diblock copolymer template and modifying the surface of NP-TiO(2) with ionic liquids (ILs), bmim-BF(4) and benmim-Cl. The effect of the molecular weight of PS-b-P4VP on the morphology of the NP-TiO(2) and IL-modified NP-TiO(2) are characterized by scanning electron microscopy and contact angle measurements. Subsequently, hybrid solar cells are fabricated using MEH-PPV and NP-TiO(2), and the effect of IL layers and IL concentrations on device performances are evaluated under AM 1.5 G illumination. The devices containing bmim-BF(4) and benmim-Cl show drastically enhanced open circuit voltages (V(oc)) of 1.05 V and 0.91 V, respectively, while the reference device without an IL layer exhibits a V(oc) of 0.60 V. Significantly improved V(oc) can be attributed to the change in interfacial energy levels by formation of ionic double layers at the TiO(2)/IL and at the IL/MEH-PPV interfaces. We also observed the trend that short circuit current decreased and V(oc) increased with increasing IL concentration, which is ascribed to interruption of charge transfer from MEH-PPV to TiO(2) and the change in interfacial energy level by shifting the vacuum level, respectively.close121