With the fast-growing e-commerce industry, online cloth shopping has become increasingly popular across the world, and in particular a large portion of the transactions happen on mobile devices. One of the biggest challenges that e-commerce companies face for selling clothes is to help customers establish realistic expectations of the garments’ ﬁt and look on their bodies via virtual ﬁtting. Not only can virtual ﬁtting help reduce logistical costs in returns, shipping and restocking, but also increase customer satisfaction rate leading to more future sales. We believe that the key to addressing the virtual ﬁtting problem is to combine deep learning with physics-based models. This requires more than the hybridization of state-of-the-art technologies, but new ideas at the interface between these disparate ﬁelds.