发新帖

DeepVariant安装出现的问题

[复制链接]
1414 8

快来加入 TensorFlowers 大家庭!

您需要 登录 才可以下载或查看,没有帐号?加入社区

x
本帖最后由 Simon0808 于 2018-4-25 11:09 编辑

         我是一名BGI的工程师。我在安装DeepVariant的时候,遇到一些问题。我希望在Tensorflow讨论区寻找一些帮助。DeepVariant的安装脚本需要安装Tensorflow,CUDA,CUDNN以及bazel等依赖的软件工具包。安装DeepVariant时,需要先运行build-prereq.sh脚本用于安装DeepVariant的配置环境,build-prereq.sh脚本分别调用了seeting.sh以及run-prereq.sh用于设置安装包的版本以及安装Tensorflow、CUDA与CUDNN等依赖包,buid-prereq.sh运行起来没有遇到问题。但是在运行build_and_test.sh测试脚本时,我遇到了一些问题。build_and_test.sh脚本调用bazel对DeepVariant进行编译,问题出现的原因我认为就是由于bazel调用有问题。安装DeepVariant遇到的问题主要有两个:                1.我在一台远程的服务器上安装DeepVariant时,在该服务器中安装的bazel是最新的0.12版本,运行buil_and_test.sh出现如下的错误:
         如果在我自己的home目录上调用bazel,是可以调用的,如下图1所示。
         图1.png
       但是进入DeepVariant的安装包路径/home/lujianliang/deepvariant-r0.6,bazel是找不到的,而且运行./build_and_test.sh也是出现同样的报错,如下图2,3和4所示。
       图2.png
       图3.png
       图4.png
       2.我的同事在另外一台本地的服务器也尝试安装DeepVariant时,也是在运行最后的测试脚本build_and_test.sh遇到的问题。我们尝试0.6版本的DeepVariant0.5版本的DeepVariant,都遇到同样的build_and_test.sh调用bazel的问题。我们使用0.6版本的DeepVariantbuild_and_test.sh测试,其中bazel的版本是0.11.0,出现如下的问题:
(14:19:27) ERROR: /home/suanfa/Downloads/deepvariant-r0.6/third_party/nucleus/io/BUILD:400:1: no such package '@com_google_absl//absl/memory': The repository could not be resolved and referenced by '//third_party/nucleus/io:vcf_writer'
(14:19:27) ERROR: Analysis of target '//deepvariant/labeler:labeled_examples_to_vcf' failed; build aborted: no such package '@com_google_absl//absl/memory': The repository could not be resolved
(14:19:27) INFO: Elapsed time: 33.556s(14:19:27) FAILED: Build did NOT complete successfully (53 packages loaded)
(14:19:27) ERROR: Couldn't start the build. Unable to run tests
          附件图5.png
   我们使用0.5版本的DeepVariantbuild_and_test.sh测试,其中bazel的版本是0.9.0,出现如下的问题:
(14:21:41) INFO: Current date is 2018-04-19
(14:21:44) ERROR: /home/suanfa/Downloads/deepvariant-r0.5/deepvariant/core/BUILD:126:1: no such package '@com_google_absl//absl/memory': The repository could not be resolved and referenced by '//deepvariant/core:vcf_writer'
(14:21:44) ERROR: Analysis of target '//deepvariant/core:vcf_writer' failed; build aborted: no such package '@com_google_absl//absl/memory': The repository could not be resolved
(14:21:44) INFO: Elapsed time: 3.102s
(14:21:44) FAILED: Build did NOT complete successfully (21 packages loaded)
    currently loading: @protobuf_archive// ... (3 packages)
(14:21:44) ERROR: Couldn't start the build. Unable to run tests
         附件图6.png
        我之前没有用过bazel,对bazel的工作原理不太熟悉。希望讨论区有人可以提供一些帮助,感谢!

build-prereq.sh、settings.sh、run-prereq.sh以及build_and_test.sh的具体脚本如下:

#以下是build-prereq.sh脚本
#!/bin/bash
set -euo pipefail

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

echo ========== Load config settings.

source settings.sh

################################################################################
# Misc. setup
################################################################################

note_build_stage "Install the runtime packages"

./run-prereq.sh

note_build_stage "Update package list"

sudo -H apt-get -qq -y update

note_build_stage "Install development packages"

sudo -H apt-get -y install pkg-config zip g++ zlib1g-dev unzip curl git

################################################################################
# Java
################################################################################

note_build_stage "Install Java and friends"

# Java is available on Kokoro, so we add this cutout.
if ! java -version 2>&1 | fgrep "1.8"; then
  echo "No Java 8, will install."
  sudo -H apt-get install -y software-properties-common debconf-utils
  # Debian needs authentication.
  # ([url=http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html]http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html[/url])
  [[ $(lsb_release -d | grep 'Debian') ]] && \
    sudo -H apt-get install -y gnupg dirmngr && \
    sudo -H apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
  sudo add-apt-repository -y ppa:webupd8team/java
  sudo -H apt-get -qq -y update
  echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | sudo debconf-set-selections
  sudo -H apt-get install -y oracle-java8-installer
  sudo -H apt-get -y install ca-certificates-java
  sudo update-ca-certificates -f
else
  echo "Java 8 found, will not reinstall."
fi

################################################################################
# bazel
################################################################################

note_build_stage "Install bazel"

function update_bazel_linux {
  BAZEL_VERSION=$1
  rm -rf ~/bazel
  mkdir ~/bazel

  pushd ~/bazel
  curl -L -O [url=https://github.com/bazelbuild/bazel/releases/download/]https://github.com/bazelbuild/bazel/releases/download/[/url]"${BAZEL_VERSION}"/bazel-"${BAZEL_VERSION}"-installer-linux-x86_64.sh
  chmod +x bazel-*.sh
  ./bazel-"${BAZEL_VERSION}"-installer-linux-x86_64.sh --user
  rm bazel-"${BAZEL_VERSION}"-installer-linux-x86_64.sh
  popd

  PATH="$HOME/bin:$PATH"
}

bazel_ver="0.11.0"
if
  v=$(bazel --bazelrc=/dev/null --nomaster_bazelrc version) &&
  echo "$v" | awk -v b="$bazel_ver" '/Build label/ { exit ($3 != b)}'
then
  echo "Bazel $bazel_ver already installed on the machine, not reinstalling"
else
  update_bazel_linux "$bazel_ver"
fi

################################################################################
# CLIF
################################################################################

note_build_stage "Install CLIF binary"

if [[ -e /usr/local/clif/bin/pyclif ]];
then
  echo "CLIF already installed."
else
  # Figure out which linux installation we are on to fetch an appropriate
  # version of the pre-built CLIF binary. Note that we only support now Ubuntu
  # 14 and 16.
  case "$(lsb_release -d)" in
    *Ubuntu*16.*.*) export DV_PLATFORM="ubuntu-16" ;;
    *Ubuntu*14.*.*) export DV_PLATFORM="ubuntu-14" ;;
    *Debian*9.*)    export DV_PLATFORM="debian" ;;
    *Debian*rodete) export DV_PLATFORM="debian" ;;
    *) echo "CLIF is not installed on this machine and a prebuilt binary is not
unavailable for this platform. Please install CLIF at
[url=https://github.com/google/clif]https://github.com/google/clif[/url] before continuing."
    exit 1
  esac

  OSS_CLIF_CURL_ROOT="${DV_PACKAGE_CURL_PATH}/oss_clif"
  OSS_CLIF_PKG="oss_clif.${DV_PLATFORM}.latest.tgz"

  if [[ ! -f "/tmp/${OSS_CLIF_PKG}" ]]; then
    curl "${OSS_CLIF_CURL_ROOT}/${OSS_CLIF_PKG}" > /tmp/${OSS_CLIF_PKG}
  fi

  (cd / && sudo tar xzf "/tmp/${OSS_CLIF_PKG}")
  sudo ldconfig  # Reload shared libraries.
fi

################################################################################
# TensorFlow
################################################################################

note_build_stage "Download and configure TensorFlow sources"

(cd .. &&
 git clone [url=https://github.com/tensorflow/tensorflow]https://github.com/tensorflow/tensorflow[/url] &&
 cd tensorflow &&
 echo | ./configure)

note_build_stage "build-prereq.sh complete"


#以下是settings.sh脚本

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

# Source this file---these options are needed for TF config and for
# successive bazel runs.

# Set this to 1 if the system image already has TensorFlow preinstalled.  This
# will skip the installation of TensorFlow.
export DV_USE_PREINSTALLED_TF="${DV_USE_PREINSTALLED_TF:-0}"

export TF_CUDA_CLANG=0
export TF_ENABLE_XLA=0
export TF_NEED_CUDA=0
export TF_NEED_GCP=1
export TF_NEED_GDR=0
export TF_NEED_HDFS=0
export TF_NEED_JEMALLOC=0
export TF_NEED_MKL=0
export TF_NEED_MPI=0
export TF_NEED_OPENCL=0
export TF_NEED_OPENCL_SYCL=0
export TF_NEED_S3=0
export TF_NEED_VERBS=0

# Used if TF_NEED_CUDA=1
export TF_CUDA_VERSION="8.0"
export CUDA_TOOLKIT_PATH="/usr/local/cuda"
export TF_CUDNN_VERSION="6"
export CUDNN_INSTALL_PATH="/usr/lib/x86_64-linux-gnu"

# Path to the public bucket containing DeepVariant-related artifacts.
export DEEPVARIANT_BUCKET="gs://deepvariant"
export DV_PACKAGE_BUCKET_PATH="${DEEPVARIANT_BUCKET}/packages"
export DV_PACKAGE_CURL_PATH="https://storage.googleapis.com/deepvariant/packages"

# Set this to 1 to use DeepVariant with GPUs. Set it to an already existing
# value in the environment (allowing command line control of the build),
# defaulting to 0 (CPU only build).
export DV_GPU_BUILD="${DV_GPU_BUILD:-0}"

# If this variable is set to 1, DeepVariant will use a TensorFlow wheel file
# compiled to use AVX and SSE instructions. This instructions require Sandy
# Bridge or better chipsets on the host machine. The default TensorFlow wheel
# files don't contain these instructions (and thereby run on a broader set of
# CPUs). Using this optimized wheel reduces the runtime of DeepVariant's
# call_variants step by ~20%. This is called the GCP (Google Cloud Platform)
# optimized wheel because all GCP instances have at least Sandy Bridge or better
# chipsets, so this wheel should run anywhere on GCP.
export DV_USE_GCP_OPTIMIZED_TF_WHL="${DV_USE_GCP_OPTIMIZED_TF_WHL:-1}"
export GCP_OPTIMIZED_TF_WHL_FILENAME="tensorflow-1.4.1.deepvariant_gcp-cp27-none-linux_x86_64.whl"
export GCP_OPTIMIZED_TF_WHL_PATH="${DV_PACKAGE_BUCKET_PATH}/tensorflow"
export GCP_OPTIMIZED_TF_WHL_CURL_PATH="${DV_PACKAGE_CURL_PATH}/tensorflow"

# Set this to 1 to use the nightly (latest) build of TensorFlow instead of a
# named release version. Set it to an already existing value in the environment
# (allowing command line control of the build), defaulting to 0 (release build).
export DV_TF_NIGHTLY_BUILD="${DV_TF_NIGHTLY_BUILD:-0}"

# Set this to 1 to make our prereq scripts install the CUDA libraries.
# If you already have CUDA installed, such as on a properly provisioned
# Docker image, it shouldn't be necessary.
export DV_INSTALL_GPU_DRIVERS="${DV_INSTALL_GPU_DRIVERS:-0}"

export PYTHON_BIN_PATH=$(which python)
export USE_DEFAULT_PYTHON_LIB_PATH=1
export DV_COPT_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-O3"

function note_build_stage {
  echo "========== [$(date)] Stage '${1}' starting"
}


#以下是run-prereq.sh脚本

#!/bin/bash

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

# This installs all the libraries (python, dso, etc) that are needed
# by DeepVariant at runtime (except for tensorflow, which is special).
# Some extra stuff may also be included.

set -euo pipefail

echo ========== Load config settings.

source settings.sh

################################################################################
# misc setup
################################################################################

note_build_stage "Misc setup"

if [[ "$EUID" = "0" ]]; then
  # Ensure sudo exists, even if we don't need it.
  apt-get update
  apt-get -y install sudo
fi

note_build_stage "Update package list"

sudo -H apt-get -qq -y update

note_build_stage "Install development packages"

sudo -H apt-get -y install pkg-config zip zlib1g-dev unzip curl

note_build_stage "Install python packaging infrastructure"

sudo -H apt-get -y install python-dev python-pip python-wheel
sudo -H pip install --upgrade pip

################################################################################
# python packages
################################################################################

note_build_stage "Install python packages"

sudo -H pip install contextlib2
sudo -H pip install enum34
sudo -H pip install intervaltree
sudo -H pip install 'mock>=2.0.0'
sudo -H pip install 'numpy==1.12'
sudo -H pip install 'requests>=2.18'
sudo -H pip install 'scipy==1.0'
sudo -H pip install 'oauth2client>=4.0.0'
sudo -H pip install 'crcmod>=1.7'
sudo -H pip install six

################################################################################
# TensorFlow
################################################################################

note_build_stage "Install TensorFlow pip package"

if [[ "${DV_USE_PREINSTALLED_TF}" = "1" ]]; then
  echo "Skipping TensorFlow installation at user request; will use pre-installed TensorFlow."
else
  # Also pip install the latest TensorFlow with cpu support. We don't build the
  # full TF from source, but instead using prebuilt version. However, we still
  # need the full source version to build DeepVariant.

  # Gets the nightly TF build: https://pypi.python.org/pypi/tf-nightly which is
  # necessary right now if we aren't pinning the TF source. We have observed
  # runtime failures if there's too much skew between the released TF package and
  # the source.
  if [[ "${DV_TF_NIGHTLY_BUILD}" = "1" ]]; then
    if [[ "${DV_GPU_BUILD}" = "1" ]]; then
      echo "Installing GPU-enabled TensorFlow nightly wheel"
      sudo -H pip install --upgrade tf_nightly_gpu
    else
      echo "Installing CPU-only TensorFlow nightly wheel"
      sudo -H pip install --upgrade tf_nightly
    fi
  else
    # Use the official TF release pip package.
    if [[ "${DV_GPU_BUILD}" = "1" ]]; then
      echo "Installing GPU-enabled TensorFlow wheel"
      sudo -H pip install --upgrade 'tensorflow-gpu==1.4'
    elif [[ "${DV_USE_GCP_OPTIMIZED_TF_WHL}" = "1" ]]; then
      echo "Installing Google Cloud Platform optimized CPU-only TensorFlow wheel"
      curl "${GCP_OPTIMIZED_TF_WHL_CURL_PATH}/${GCP_OPTIMIZED_TF_WHL_FILENAME}" \
        > "/tmp/${GCP_OPTIMIZED_TF_WHL_FILENAME}"
      sudo -H pip install --upgrade "/tmp/${GCP_OPTIMIZED_TF_WHL_FILENAME}"
    else
      echo "Installing standard CPU-only TensorFlow wheel"
      sudo -H pip install --upgrade 'tensorflow==1.4'
    fi
  fi
fi


################################################################################
# CUDA
################################################################################

if [[ "${DV_GPU_BUILD}" = "1" ]]; then
  if [[ "${DV_INSTALL_GPU_DRIVERS}" = "1" ]]; then
    if [[ "$(lsb_release -d)" != *Ubuntu*16.*.* ]]; then
      echo "CUDA installation only configured for Ubuntu 16"
      exit 1
    fi

    # from https://cloud.google.com/compute/docs/gpus/add-gpus
    echo "Checking for CUDA..."
    if ! dpkg-query -W cuda-8-0; then
      echo "Installing CUDA..."
      CUDA_DEB="cuda-repo-ubuntu1604_8.0.61-1_amd64.deb"
      curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/${CUDA_DEB}
      sudo -H dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
      sudo -H apt-get update
      sudo -H apt-get -y install cuda-8-0
    fi

    echo "Checking for CUDNN..."
    if [[ ! -e /usr/local/cuda-8.0/include/cudnn.h ]]; then
      echo "Installing CUDNN..."
      CUDNN_TAR_FILE="cudnn-8.0-linux-x64-v6.0.tgz"
      wget http://developer.download.nvidia.com/compute/redist/cudnn/v6.0/${CUDNN_TAR_FILE}
      tar -xzvf ${CUDNN_TAR_FILE}
      sudo cp -P cuda/include/cudnn.h /usr/local/cuda-8.0/include
      sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-8.0/lib64/
      sudo chmod a+r /usr/local/cuda-8.0/lib64/libcudnn*
      sudo ldconfig
    fi

    # Tensorflow says to do this.
    sudo -H apt-get -y install libcupti-dev
  fi

  # If we are doing a gpu-build, nvidia-smi should be install. Run it so we
  # can see what gpu is installed.
  nvidia-smi || :
fi


################################################################################
# Misc dependencies
################################################################################

note_build_stage "Install other packages"

# for htslib
sudo -H apt-get -y install libssl-dev libcurl4-openssl-dev liblz-dev libbz2-dev liblzma-dev

# for the debruijn graph
sudo -H apt-get -y install libboost-graph-dev

note_build_stage "run-prereq.sh complete"


#以下是build_and_test.sh脚本

#!/bin/bash

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

# NOLINT
set -eux -o pipefail

source settings.sh

# Run all deepvariant tests.  Take bazel options from args, if any.
# Note: If running with GPU, tests must be executed serially due to a GPU
# contention issue.
if [[ "${DV_GPU_BUILD:-0}" = "1" ]]; then
  bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@" \
    deepvariant/...
  # Also run the GPU-specific tests.
  bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@" \
    deepvariant:gpu_tests
else
  # Running parallel tests on CPU.
  bazel test -c opt ${DV_COPT_FLAGS} "$@" deepvariant/...
fi

# Create some build artifacts.
bazel build -c opt ${DV_COPT_FLAGS} "$@" deepvariant:binaries

# Bundle the licenses.
bazel build :licenses_zip

我知道答案 回答被采纳将会获得10 金币 + 10 金币 已有8人回答
本楼点评(0) 收起

精彩评论8

舟3332  TF芽芽  发表于 2018-4-25 17:40:51 | 显示全部楼层
本帖最后由 舟3332 于 2018-4-25 18:06 编辑

不是 bazel 专家,不过这个错误看着好像是有依赖无法找到的问题

(14:19:27) ERROR: /home/suanfa/Downloads/deepvariant-r0.6/third_party/nucleus/io/BUILD:400:1: no such package '@com_google_absl//absl/memory': The repository could not be resolved and referenced by '//third_party/nucleus/io:vcf_writer'
(14:19:27) ERROR: Analysis of target '//deepvariant/labeler:labeled_examples_to_vcf' failed; build aborted: no such package '@com_google_absl//absl/memory': The repository could not be resolved
(14:19:27) INFO: Elapsed time: 33.556s(14:19:27) FAILED: Build did NOT complete successfully (53 packages loaded)
(14:19:27) ERROR: Couldn't start the build. Unable to run tests

@com_google_absl//absl/memory 这个包要能解析,首先 bazel 要知道 com_google_absl 能够解析到哪一个 github repo

这个 mapping 一般是在 workspace 里面给出的,也就是 https://github.com/google/deepvariant/blob/r0.6/WORKSPACE 这个文件
# Abseil libraries
git_repository(
    name = "com_google_absl_py",
    # redacted
    commit = "5e343642d987268df199b4c851b7dd3d687ac316",
    remote = "https://github.com/abseil/abseil-py.git",
)
# Note: com_google_absl (the C++ abseil library) is provided by TensorFlow.

可以看到,其中定义了 com_google_absl_py 这个名称,但是 com_google_absl 是从 tensorflow 载入的。

我们继续往下看,可以看到 bazel 将会载入 tensorflow 目录的 tensorflow/workspace.bzl 并将其用作 org_tensorflow
# Import all of the tensorflow dependencies.
load("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")

# Import tensorflow.  Note path.
local_repository(
    name = "org_tensorflow",
    path = "../tensorflow",
)


那么我们看一下 tensorflow/workspace.bzl 中是否对 com_google_absl 有定义。

果然 https://github.com/tensorflow/te ... rflow/workspace.bzl 定义了这个名称
  tf_http_archive(
      name = "com_google_absl",
      urls = [
          "https://mirror.bazel.build/github.com/abseil/abseil-cpp/archive/720c017e30339fd1786ce4aac68bc8559736e53f.tar.gz",
          "https://github.com/abseil/abseil-cpp/archive/720c017e30339fd1786ce4aac68bc8559736e53f.tar.gz",
      ],
     sha256 = "5996380e3e8b981f55d1c8d58e709c00dbb4806ba367be75d0925a68cc2f6478",
     strip_prefix = "abseil-cpp-720c017e30339fd1786ce4aac68bc8559736e53f",
     build_file = clean_dep("//third_party:com_google_absl.BUILD"),
  )


理清了思路回头看一下这个问题。我猜测很有肯能是 bazel 没有在 tensorflow/workspace.bzl 里面看到 com_google_absl

建议可以看一下父目录是否有 tensorflow 的源代码。

如果没有的话也可以通过这个脚本生成 https://github.com/google/deepvariant/blob/r0.6/build-prereq.sh
(cd .. &&
 git clone [url=https://github.com/tensorflow/tensorflow]https://github.com/tensorflow/tensorflow[/url] &&
 cd tensorflow &&
 git checkout v1.7.0 &&
 echo | ./configure)



本楼点评(0) 收起
舟3332  TF芽芽  发表于 2018-4-25 18:12:37 | 显示全部楼层
另外关于 1) 我觉得可能是 bazel 12 和 deepvariant 不兼容。Deepvariant 自带的 bazel 也是 11 版本。
本楼点评(3) 收起
  • Simon0808我已经尝试卸载bazel 12,然后安装bazel 11,也是不行。另外关于问题2),我同事是已经运行脚本上build-prereq.sh,已经在DeepVariant-r0.6的父目录建立tensorflow的源代码。我出现的问题就是找不到父目录上的Tensorflow目录中的"/home/lujianliang/deepvariant-r0.6/../tensorflow/.tf_configure.bazelrc"。
    2018-4-26 20:37 回复
  • 舟3332回复 Simon0808 : 你用的是哪个版本? deepvariant master branch 上的最新版吗?
    2018-4-28 07:13 回复
  • Simon0808我使用的是DeepVariant的0.6版本。我发现一个问题:DeepVariant0.6版本的源代码中的run-prereq.sh是安装TensorflowGPU1.4版本的,cuda8.0,cudnn v6。我的服务器上也是安装这些版本。但是build-prereq.sh这个脚本的最后一段代码是先下载Tensorflow的源代码,然后调用"git checkout v1.7.0"来构建最新的Tensorflow1.7版本。那么如果构建Tensorflow GPU1.7版本,Tensorflow的官网介绍是需要cuda9.0以及cudnn v7的吧。代码是“(cd ..
    2018-4-29 08:43 回复
pichuan  TF荚荚  发表于 2018-4-26 03:01:30 | 显示全部楼层
Hi, thanks for your question.
你觀察到的和前面的回答都提到重點: DeepVariant 0.6 currently works with Bazel version 0.11.0, but not 0.12.0.
build-prereq.sh 裡面設定 bazel 版本為 0.11.0, 所以理論上如果你先跑過build-prereq.sh 這個 script, 他應該會安裝0.11.0

你有試過先執行:
build-prereq.sh
然後再用 build_and_test.sh 嗎?

在bazel版本為0.11.0的狀況之下,是否可以build呢?

我目前測試的環境是使用 Ubuntu 16.04, 基本上直接使用 Case Study 建議的設定從 GCP 上取得一個 Ubuntu 16.04 的機器:
https://github.com/google/deepva ... riant-case-study.md
在這樣的環境下,如果我 clone DeepVariant 的codebase, 然後跑
./build-prereq.sh
./build_and_test.sh
的話,是可以build的。不過我也測試了:如果我將 bazel 升級到 0.12.0, 就會產生錯誤。

我會再研究看看如何在未來版本裡upgrade bazel version. Thanks for your feeback!!!
本楼点评(3) 收起
  • Simon0808感谢您给出的帮助。我已经尝试卸载bazel 12,重新安装bazel 11。但是还是出现了同样的问题:找不到父目录上的Tensorflow目录中的"/home/lujianliang/deepvariant-r0.6/../tensorflow/.tf_configure.bazelrc"。同时注意到你的一句话“基本上直接使用 Case Study 建議的設定從 GCP 上取得一個 Ubuntu 16.04 的機器”,我理解您的意思是从GCP下载取得那些已经编译好zip文件运行DeepVariant的三个步骤吗?其实我们是已经可以从GCP下载make_examples.zip,call_variants.zip和postprocess_variants.zip这个三个zip文件对BAM文件进行变异检测了。我们的问题是如果我们想修改DeepVariant的代码,然后是不是需要通过bazel对py脚本进行编译才能生成类似的zip文件然后对bam文件进行变异检测。我们已经试过直接使用“python make_examples.py <options>”这样方式运行,但是会出现一些缺少依赖库的报错,所以我们在想是跟bazel没有完全安装或者bazel的使用方法不正确所造成的。希望把bazel给安装准确。
    2018-4-26 21:31 回复
  • 舟3332回复 Simon0808 : pichuan 应该说的是直接从 GCP 上开 Ubuntu 的虚拟机跑没有问题。
    2018-4-28 07:14 回复
  • pichuan回复 Simon0808 :
    Hi,
    I just released a new update:
    https://github.com/google/deepvariant/releases/tag/v0.6.1

    which hopefully fixes your build problem on Debian. (I tested it on a Debian machine and it builds for me now)

    Can you give it a try and see if you can build it on Debian now?
    2018-5-1 12:12 回复
舟3332  TF芽芽  发表于 2018-4-26 07:25:10 | 显示全部楼层
pichuan 发表于 2018-4-26 03:01
Hi, thanks for your question.
你觀察到的和前面的回答都提到重點: DeepVariant 0.6 currently works with ...

哇~ DeepVariant 开发团队也入驻了。
本楼点评(1) 收起
smile  TF豆豆  发表于 2018-4-26 10:40:20 | 显示全部楼层
哇,欢迎大牛入住
本楼点评(0) 收起
Simon0808  TF荚荚  发表于 2018-4-29 08:47:20 | 显示全部楼层
我使用的是DeepVariant的0.6版本。我发现一个问题:DeepVariant0.6版本的源代码中的run-prereq.sh是安装TensorflowGPU1.4版本的,cuda8.0,cudnn v6。我的服务器上也是安装这些版本。但是build-prereq.sh这个脚本的最后一段代码是先下载Tensorflow的源代码,然后调用"git checkout v1.7.0"来构建最新的Tensorflow1.7版本。那么如果构建Tensorflow GPU1.7版本,Tensorflow的官网介绍是需要cuda9.0以及cudnn v7的吧。代码是“(cd .. &&git clone https://github.com/tensorflow/tensorflow &&cd tensorflow && git checkout v1.7.0 && echo | ./configure)”。另外一个与我一起尝试安装DeepVariant的中科大的人员使用的是tensorflow 1.6版本,他发现Tensorflow1.4版本和1.7版本的workspace有很大差别,1.6版本跟1.7版本较为接近,他使用Tensorflow 1.6版本以及通过build_and_test.sh的33个test中10个test。因为他的机器也是安装cuda8.0以及cudnn v6,Tensorflow 1.7版本也不能使用。
本楼点评(0) 收起
M丶Sulayman  TF豆豆  发表于 2018-4-29 09:49:19 | 显示全部楼层
工程师啊~~~厉害了~~~欢迎~~~
本楼点评(0) 收起
舟3332  TF芽芽  发表于 2018-5-1 10:49:01 | 显示全部楼层
Simon0808 发表于 2018-4-29 08:47
我使用的是DeepVariant的0.6版本。我发现一个问题:DeepVariant0.6版本的源代码中的run-prereq.sh是安装Ten ...

这段我也没看懂。

我猜测可能是 deepvariant 程序在跑的时候会去调用系统里面用的 TF 版本,也就是 1.4
编译的时候可能会需要 TF 的一些头文件和依赖所以需要 tf 的源代码。
如果是这样的话大概两个版本不太一致也不是太大的问题

我在 Google Compute Engine 上边抓了一个 Debian 的机器测试了一下,这个版本不一致应该不是主要问题。

有些测试无法通过的情况可以贴一下错误信息大家一起看一下~
本楼点评(0) 收起
您需要登录后才可以回帖 登录 | 加入社区

本版积分规则

快速回复 返回顶部 返回列表