09_Up and Running with TensorFlow_Anaconda3-4.2.0_No module named joblib_KeyError 0_Normal Equation

     TensorFlow is a powerful open source software library for numerical computation, particularly well suited and fine-tuned對進行微調 for large-scale Machine Learning. Its basic principle is simple: you first define in Python a graph of computations to perform (for example, the one in Figure 9-1), and then TensorFlow takes that graph and runs it efficiently using optimized C++ code.

Figure 9-1. A simple computation graph

     Most importantly, it is possible to break up the graph into several chunks塊 and run them in parallel across multiple CPUs or GPUs (as shown in Figure 9-2). TensorFlow also supports distributed computing, so you can train colossal龐大的 neural networks on humongous極大的 training sets in a reasonable amount of time by splitting the computations across hundreds of servers (see Chapter 12). TensorFlow can train a network with millions of parameters on a training set composed of billions of instances with millions of features each. This should come as no surprise, since TensorFlow was
developed by the Google Brain team and it powers many of Google’s large-scale services, such as Google Cloud Speech, Google Photos, and Google Search.

Figure 9-2. Parallel computation on multiple CPUs/GPUs/servers

     When TensorFlow was open-sourced in November 2015, there were already many popular open source libraries for Deep Learning (Table 9-1 lists a few), and to be fair most of TensorFlow's features already existed in one library or another. Nevertheless, TensorFlow's clean design, scalability可擴展性, flexibility, and great documentation (not to mention Google's name) quickly boosted it to the top of the list. In short, TensorFlow was designed to be flexible, scalable, and production-ready, and existing frameworks arguably hit only two out of the three of these. Here are some of TensorFlow's highlights:

• It runs not only on Windows, Linux, and macOS, but also on mobile devices, including both iOS and Android.
  It provides a very simple Python API called TF.Learn (tensorflow.con trib.learn)###Not to be confused with the TFLearn library, which is an independent project.###, compatible with Scikit-Learn. As you will see, you can use it to train various types of neural networks in just a few lines of code. It was previously
  an independent project called Scikit Flow (or skflow).
• It also provides another simple API called TF-slim (tensorflow.contrib.slim) to simplify building, training, and evaluating neural networks.
   
• Several other high-level APIs have been built independently on top of Tensor‐Flow, such as Keras or Pretty Tensor.
   
• Its main Python API offers much more flexibility (at the cost of higher complexity) to create all sorts of computations, including any neural network architecture you can think of.
   
• It includes highly efficient C++ implementations of many ML operations, particularly those needed to build neural networks. There is also a C++ API to define your own high-performance operations.
   
• It provides several advanced optimization nodes to search for the parameters that minimize a cost function. These are very easy to use since TensorFlow automatically takes care of computing the gradients of the functions you define. This is called automatic differentiating自動分解 (or autodiff).
   
• It also comes with a great visualization tool called TensorBoard that allows you to browse through the computation graph, view learning curves, and more.
   
• Google also launched a cloud service to run TensorFlow graphs.
   
• Last but not least, it has a dedicated team of passionate and helpful developers, and a growing community contributing to improving it. It is one of the most popular open source projects on GitHub, and more and more great projects are being built on top of it (for examples, check out the resources page on https://www.tensorflow.org/, or https://github.com/jtoy/awesome-tensorflow). To ask technical questions, you should use http://stackoverflow.com/ and tag your question with "tensorflow". You can file bugs and feature requests through GitHub. For general discussions, join the Google group.

     In this chapter, we will go through the basics of TensorFlow, from installation to creating, running, saving, and visualizing simple computational graphs. Mastering these basics is important before you build your first neural network (which we will do in the next chapter).

Table 9-1. Open source Deep Learning libraries (not an exhaustive list)

Installation

Anaconda3-4.2.0-Windows-x86_64
link：https://pan.baidu.com/s/1dTBmUZdqxHpcum6MQ6RAuA 
code：62hx

In Anaconda prompt(Anaconda3)

1. type command: conda create --name tensorflow python=3.5.2
2. type command: conda activate tensorflow
   type command: conda info --envs
   type command: python --version
   type command: python
   type command: import tensorflow as tf
   
   type command: conda list
   
   Solution:
   type command: conda install tensorflow
   
   type command: conda list
   
3. if you want to update scikit-learn

   type command: conda update scikit-learn

4.  

   Open your Jupyter notebook: type the following codes and run

    

   import tensorflow;
   print(tensorflow.__version__)

       

Creating Your First Graph and Running It in a Session

The following code creates the graph represented in Figure 9-1:

x = tf.Variable(3, name='x')
y = tf.Variable(4, name='y')
f = x*x*y+y+2

f

 

     That's all there is to it! The most important thing to understand is that this code does not actually perform any computation, even though it looks like it does (especially the last line). It just creates a computation graph. In fact, even the variables are not initialized yet. To evaluate this graph, you need to open a TensorFlow session and use it to initialize the variables and evaluate f. A TensorFlow session takes care of placing the operations onto devices such as CPUs and GPUs and running them, and it holds all the variable values. 

     The following code creates a session, initializes the variables, and evaluates, and f then closes the session (which frees up resources):

sess = tf.Session()     #creates a session
sess.run(x.initializer) #initializes the variables
sess.run(y.initializer)
result = sess.run(f)    #evaluates f
print(result)

sess.close() # closes the session (which frees up resources)

     Having to repeat sess.run() all the time is a bit cumbersome麻煩的, but fortunately there is a better way:

with tf.Session() as sess: # the session is set as the default session
    x.initializer.run()   #==tf.get_default_session().run(x.initializer)
    y.initializer.run()   
    result = f.eval()     #==tf.get_default_session().run(f)
    #the session is automatically closed at the end of the block

result

     Inside the with block, the session is set as the default session. Calling x.initial izer.run() is equivalent to calling tf.get_default_session().run(x.initializer), and similarly f.eval() is equivalent to calling tf.get_default_session().run(f). This makes the code easier to read. Moreover, the session is automatically closed at the end of the block.

     Instead of manually running the initializer for every single variable, you can use the global_variables_initializer() function. Note that it does not actually perform the initialization immediately, but rather creates a node in the graph that will initialize all variables when it is run:

x = tf.Variable(3, name='x')
y = tf.Variable(4, name='y')
f = x*x*y+y+2

#it does not actually perform the initialization immediately
init = tf.global_variables_initializer() #creates a node in the graph that will initialize all variables

sess = tf.InteractiveSession()
init.run()                               #when it is run
result = f.eval()
print(result)

     Inside Jupyter or within a Python shell you may prefer to create an InteractiveSession. The only difference from a regular Session is that when an InteractiveSession is created it automatically sets itself as the default session, so you don't need a with block (but you do need to close the session manually when you are done with it):

sess.close()

result

     A TensorFlow program is typically split into two parts:

1. the first part builds a computation graph (this is called the construction phase),
   The construction phase typically builds a computation graph representing the ML model and the computations required to train it.
2. and the second part runs it (this is the execution phase).
   The execution phase generally runs a loop that evaluates a training step repeatedly (for example, one
   step per mini-batch), gradually improving the model parameters. We will go through an example shortly.

Managing Graphs

     Any node you create is automatically added to the default graph:

https://docs.w3cub.com/tensorflow~python/tf/reset_default_graph/

tf.reset_default_graph()
     Clears the default graph stack and resets the global default graph.

     NOTE: The default graph is a property of the current thread. This function applies only to the current thread. Calling this function while a tf.Session or tf.InteractiveSession is active will result in undefined behavior. Using any previously created tf.Operation or tf.Tensorobjects after calling this function will result in undefined behavior.

Raises:

• AssertionError: If this function is called within a nested graph.

tf.reset_default_graph() # since sess.close()

x1 = tf.Variable(1)
x1.graph is tf.get_default_graph()

     In most cases this is fine, but sometimes you may want to manage multiple independent graphs. You can do this by creating a new Graph and temporarily making it the default graph inside a with block, like so:

graph = tf.Graph()
with graph.as_default():
    x2 = tf.Variable(2)
    
x2.graph is graph

x2.graph is tf.get_default_graph()

######################
TIP
     In Jupyter (or in a Python shell), it is common to run the same commands more than once while you are experimenting. As a result, you may end up with a default graph containing many duplicate nodes. One solution is to restart the Jupyter kernel (or the Python shell), but a more convenient solution is to just reset the default graph by running tf.reset_default_graph().
######################

Lifecycle of a Node Value

     When you evaluate a node, TensorFlow automatically determines the set of nodes that it depends on and it evaluates these nodes first. For example, consider the following code:

# First, this code defines a very simple graph.
w = tf.constant(3) #tf.constant(3): Creates a constant tensor
x = w + 2  
y = x + 5  
z = x * 3  

# Then it starts a session and runs the graph to evaluate
with tf.Session() as sess:
    print(y.eval()) # 10 # runs the graph(containing nodes[w,x,y,z]) to evaluate y # first evaluates w, then x, then y; 
    print(z.eval()) # 15 # Finally, the code runs the graph to evaluate z

     First, this code defines a very simple graph. Then it starts a session and runs the graph ###(containing nodes[w,x,y,z] and constants[2,5,3]) ### to evaluate y: TensorFlow automatically detects that y depends on x, which depends on w, so it first evaluates w, then x, then y, and returns the value of y. Finally, the code runs the graph to evaluate z. Once again, TensorFlow detects that it must first evaluate w and x. It is important to note that it will not reuse the result of the previous evaluation of w and x. In short, the preceding code evaluates w and x twice.

###

 TensorFlow operations (also called ops for short) can take any number of inputs and produce any number of outputs
y.eval()   #tell the program to run the graph to just evaluate y
z.eval()   #tell the program to run the graph to just evaluate z
###

     All node values ###[w,x,y,z]###  are dropped between graph runs ###(runs the graph to evaluate y and runs the graph to evaluate z)###, except variable values ### tf.Variable(3, name='x') ###, which are maintained by the session across graph runs (queues and readers also maintain some state, as we will see in Chapter 12). A variable starts its life when its initializer is run, and it ends when the session is closed.

     If you want to evaluate y and z efficiently, without evaluating w and x twice as in the previous code, you must ask TensorFlow to evaluate both y and z in just one graph run, as shown in the following code:

with tf.Session() as sess:
    y_val, z_val = sess.run([y,z]) #  evaluate both y and z in just one graph run
    print(y_val) # 10
    print(z_val) # 15

### 
y_val, z_val = sess.run([y,z])   #tell the program to run the graph to evaluate both y and z
###

###################################
WARNING
     In single-process TensorFlow, multiple sessions do not share any state, even if they reuse the same graph (each session would have its own copy of every variable). In distributed TensorFlow (see Chapter 12), variable state is stored on the servers, not in the sessions, so multiple sessions can share the same variables.
################################### 

Linear Regression with TensorFlow

     TensorFlow operations (also called ops for short) can take any number of inputs and produce any number of outputs. For example, the addition and multiplication ops each take two inputs and produce one output. Constants and variables take no input (they are called source ops). The inputs and outputs are multidimensional arrays, called tensors (hence the name "tensor flow"). Just like NumPy arrays, tensors have a type and a shape. In fact, in the Python API tensors are simply represented by NumPy ndarrays. They typically contain floats, but you can also use them to carry strings (arbitrary byte arrays).

     In the examples so far, the tensors just contained a single scalar value, but you can of course perform computations on arrays of any shape. For example, the following code manipulates 2D arrays to perform Linear Regression on the California housing dataset (introduced in https://blog.csdn.net/Linli522362242/article/details/103387527). It starts by fetching the dataset; then it adds an extra bias input feature ( = 1) to all training instances (it does so using NumPy so it runs immediately); then it creates two TensorFlow constant nodes, X and y, to hold this data and the targets, and it uses some of the matrix operations provided by TensorFlow to define theta. These matrix functions — transpose(), matmul(), and matrix_inverse() — are self-explanatory, but as usual they do not perform any computations immediately; instead, they create nodes in the graph that will perform them when the graph is run. You may recognize that the definition of theta corresponds to the Normal Equation ( ; see https://blog.csdn.net/Linli522362242/article/details/104005906  find the value of θ ==> minimizes MSE cost function ==> minimizes the RMSE ). Finally, the code creates a session and uses it to evaluate theta(θ).

import numpy as np
from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing()

.... 

ModuleNotFoundError: No module named 'joblib'
type command "conda list": 

... ...

... ...

type command "pip install joblib"

Note: "joblib" requires a difference Python: 3.5.6 not in ">=3.6"

Let's type command "easy_install joblib"

It seems good?
Let's go back the Jupyter notebook, 

import numpy as np
from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing()

KeyError: 0

import numpy as np
from sklearn.datasets import fetch_california_housing

help(fetch_california_housing)

      
solution 1:

Oh, no, we haven't solve the issue.
so we still have to use the original solution in which we have to write our function to download the california_housing data. https://blog.csdn.net/Linli522362242/article/details/103387527

solution 2:
Click the link:
https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html#examples-using-sklearn-datasets-fetch-california-housing

Click the link "source"
https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/datasets/_california_housing.py#L53

housing = fetch_california_housing("../datasets/housing",download_if_missing=True)
#waiting few minutes

check your folder or default directory: 

print( housing.DESCR )

# note:  the this dataset has done missing value process, Missing Attribute Values: None

import numpy as np
from sklearn.datasets import fetch_california_housing

############  It starts by fetching the dataset   ############
housing = fetch_california_housing("../datasets/housing",download_if_missing=True)
#waiting few minutes
m,n = housing.data.shape #(20640, 8)

############  it adds an extra bias input feature (intercept=1) to all training instances   ############
housing_data_plus_bias = np.c_[np.ones((m,1)), housing.data] #np.c_ for list and return an array
# OR
# housing_data_plus_bias = np.concatenate( [np.ones((m,1)), housing.data], axis=1)

###########   it creates two TensorFlow constant nodes X and y to hold this data and the targets ###########
X = tf.constant(housing_data_plus_bias, dtype=tf.float32, name="X")
# housing.target.shape : (20640,)
# housing.target.reshape(-1,1).shape : (20640, 1)
y = tf.constant(housing.target.reshape(-1,1), dtype=tf.float32, name="y")
XT = tf.transpose(X)
###transpose(), matmul()==np.dot(), and matrix_inverse() 
###the Normal Equation
theta = tf.matmul( tf.matmul( tf.matrix_inverse( tf.matmul(XT,X) ), XT ), y )
with tf.Session() as sess:
    theta_value = theta.eval()
print(theta_value)

     The main benefit of this code versus computing the Normal Equation directly using NumPy is that TensorFlow will automatically run this on your GPU card if you have one (provided you installed TensorFlow with GPU support, of course; see Chapter 12 for more details).

Compare with Scikit-Learn

from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit( housing.data, housing.target.reshape(-1,1) )

# lin_reg.intercept_.shape : (1,) -- .reshape(-1,1) --> (1,1)
# lin_reg.coef_.shape : (1, 8) --> .T --> (8,1)
print( np.r_[lin_reg.intercept_.reshape(-1,1), lin_reg.coef_.T ]) #np.r_ for list and return an array
# OR 
# print(  np.concatenate([ lin_reg.intercept_.reshape(-1,1), lin_reg.coef_.T ], axis=0) )

Implementing Gradient Descent

     Let's try using Batch Gradient Descent (introduced in https://blog.csdn.net/Linli522362242/article/details/104005906 Chapter 4) instead of the Normal Equation. First we will do this by manually computing the gradients, then we will use TensorFlow's autodiff feature to let TensorFlow compute the gradients automatically, and finally we will use a couple of TensorFlow's out-of-the-box optimizers.

###############################
WARNING
     When using Gradient Descent, remember that it is important to first normalize the input feature vectors, or else training may be much slower. You can do this using TensorFlow, NumPy, Scikit-Learn's StandardScaler, or any other solution you prefer. The following code assumes that this normalization has already been done.
###############################

09_Up and Running with TensorFlow_02

https://blog.csdn.net/Linli522362242/article/details/106290394