TensorFlowOnSpark

GitHub describes it as TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.. The repository metadata lists Python as its primary language. The metadata lists the Apache-2.0 license. This article stays within the project description and details documented in the GitHub repository README.

GitHub describes it as TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.. The repository metadata lists Python as its primary language. The metadata lists the Apache-2.0 license. The README describes the project this way: TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark clusters.

The README section "TensorFlowOnSpark" states: By combining salient features from the TensorFlow deep learning framework with Apache Spark and Apache Hadoop, TensorFlowOnSpark enables distributed deep learning on a cluster of GPU and CPU servers.

The README section "TensorFlowOnSpark" states: It enables both distributed TensorFlow training and inferencing on Spark clusters, with a goal to minimize the amount of code changes required to run existing TensorFlow programs on a shared grid. Its Spark-compatible API helps manage the TensorFlow cluster with the following steps:

The README section "TensorFlowOnSpark" states: 1. Startup - launches the Tensorflow main function on the executors, along with listeners for data/control messages. 1. Data ingestion - InputMode.TENSORFLOW - leverages TensorFlow's built-in APIs to read data files directly from HDFS. - InputMode.SPARK - sends Spark RDD data to the TensorFlow nodes via a TFNode.DataFeed class. Note that we leverage the Hadoop Input/Output Format to access TFRecords on HDFS. 1. Shutdown - shuts down the Tensorflow workers and PS nodes on the executors.

The repository README is the source for this review. It does not replace a local installation or an independent test.

• Source 1 · Project repository
• Source 2 · README