深度學習基本概念softmax、sigmoid、均方誤差、交叉熵

    在神經網絡的最後一層總用到的激活函數爲softmax和sigmoid。 在經過以上 激活函數後，線性迴歸選用均方誤差爲損失函數，邏輯迴歸選擇交叉熵爲損失函數。以下代碼使用tensorflow實現。

  softmax激活函數：用在多分類，一個輸入只能有一個輸出，輸出標籤獨立（onehot類型標籤，只有一個爲1，其他都是0）），如貓狗大戰一張圖像，識別結果非貓即狗。

---

tensorflow 實現 tf.nn.sigmoid(logits)，自己定義實現如下

def Sigmoid_( logits ):
    date = tf.cast(logits,tf.int32)
    result = []
    with tf.Session() as sess:
        date = sess.run(date)
        for i in date:
            sig = round(1/(1+math.exp(-i)),6 )
            result.append(sig)
    return result

---

 sigmoid激活函數：用在多標籤，即輸入一張圖像，可以同時多標籤種類爲1，如一張圖像即有貓又有狗標籤中含有兩個1.

 tensorflow實現tf.nn.softmax(logits)，自定義實現

def Softmax_( logits ):
    date = tf.cast(logits,tf.int32)
    result = []
    with tf.Session() as sess:
        date = sess.run(date)
        sum = 0
        for i in date:
            sum += math.exp(i)
        for i in date:
            result.append( round(math.exp(i)/sum, 6))
        return result

 

---

  softmax的交叉熵損失函數與sigmoid的略有不同，因爲softmax標籤只有一張可能性

softmax:                    

sigmoid:                  

整體實現代碼：

# coding:utf-8
import tensorflow as tf
import os
import math
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'


def Sigmoid_( logits ):
    date = tf.cast(logits,tf.int32)
    result = []
    with tf.Session() as sess:
        date = sess.run(date)
        for i in date:
            sig = round(1/(1+math.exp(-i)),6 )
            result.append(sig)
    return result


def Softmax_( logits ):
    date = tf.cast(logits,tf.int32)
    result = []
    with tf.Session() as sess:
        date = sess.run(date)
        sum = 0
        for i in date:
            sum += math.exp(i)
        for i in date:
            result.append( round(math.exp(i)/sum, 6))
        return result


def get_loss( res1 ):
    print("softmax loss")
    labels = [0, 1, 0]
    loss_sum = 0
    for i in range(len(labels)):
        loss_ = -labels[i]*tf.log(res1[i])
        loss_sum =loss_sum+loss_
    return  loss_sum


def get_sigmoid_loss( res1 ):
    print("sigmoid loss")
    labels = [0, 1, 0]
    loss_sum = []
    for i in range(len(labels)):
        loss_ = -labels[i]*tf.log(res1[i]) - (1-labels[i])*tf.log(1-res1[i])
        loss_sum.append(loss_)
    return  loss_sum

print('=====================sigmod test===============================')

logits = tf.constant([2,5,1],dtype=tf.float32)
labels = [0,1,0]
label = tf.convert_to_tensor(labels)
label = tf.cast(label,tf.float32)

res_sigmoid = tf.nn.sigmoid(logits)

ls = get_sigmoid_loss(res_sigmoid)
res2 = Sigmoid_(logits)
res3 = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits,labels=label)

with tf.Session() as sess:
    res1,res3 = sess.run([res_sigmoid,res3])
    los = sess.run(ls)
    print("tf計算結果：",   res1)
    print("手寫計算結果：",  res2)
    print("tf計算結果：",   res3)
    print("手寫計算結果：",  los)

print('====================softmax test================================')
res1 = tf.nn.softmax(logits)
ls = get_loss(res1)

res2 = Softmax_(logits)
res3 = tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits,labels=labels)
with tf.Session() as sess:
    res1,res3 = sess.run([res1,res3])
    los = sess.run(ls)
    print("tf計算結果：",   res1)
    print("手寫計算結果：", res2)
    print("tf計算結果：",   res3)
    print("手寫計算結果：",  los)

print('====================================================')