Clone and checkout the latest 1.14.
git clone https://github.com/tensorflow/tensorflow.git git checkout r1.14
Then configure the build. For some reason 1.14 picked up my cuda at /usr/local/cuda instead of the one in LD_LIBRARY_PATH. I pointed /usr/local/cuda to /usr/local/cuda-9.1. Everything seemed to get picked up ok then. **CUDA-9.1 DOESN'T WORK with TF 1.14.**
I started building using this command, with bazel 0.25.2.
bazel build //tensorflow/tools/pip_package:build_pip_package
The latest instructions suggest using "--config=cuda" and "--config=v1", I didn't use either though.
Finally, building the whl is as simple as running the created command.
/bazel-bin/tensorflow/tools/pip_package/build_pip_package ../tensorflow_pkg/
That left me with a working tensorflow, although some things have changed from 1.13.0 to 1.13.2 and 1.14.1 in the callbacks. Now training metrics are averaged while an epoch accumulates. So the last batch is probably the same as the epoch values.