Make sure we don't remove a piece that don't exist.