網站該不該給用戶貸款呢

1.數據預處理

1.1去掉Url以及描述等內容

​import pandas as pd
loans_2007 = pd.read_csv('LoanStats3a.csv', skiprows=1) #skiprows=1表示從第二行開始讀
half_count = len(loans_2007) / 2
loans_2007 = loans_2007.dropna(thresh=half_count, axis=1)#thresh=n，即剔除NA值，保留下來的每一行，其非NA的數目>=n
loans_2007 = loans_2007.drop(['desc', 'url'],axis=1)
loans_2007.to_csv('loans_2007.csv', index=False)

查看上一步處理後的數據信息

loans_2007 = pd.read_csv("loans_2007.csv")
#loans_2007.drop_duplicates()
print(loans_2007.iloc[0])
print(loans_2007.shape[1])

結果顯示總共有52個維度（圖片裏我沒有放全），接下來的問題是怎樣從這52個維度中選取機器學習要用到的維度。

1.2去掉與貸款可能無關的維度

loans_2007 = loans_2007.drop(["id", "member_id", "funded_amnt", "funded_amnt_inv", "grade", "sub_grade", "emp_title", "issue_d","zip_code", "out_prncp", "out_prncp_inv", "total_pymnt", "total_pymnt_inv", "total_rec_prncp","total_rec_int", "total_rec_late_fee", "recoveries", "collection_recovery_fee", "last_pymnt_d", "last_pymnt_amnt"], axis=1)
print(loans_2007.iloc[0])
print(loans_2007.shape[1])

1.3選擇最後是否貸款那一列的分類結果

print(loans_2007['loan_status'].value_counts())

用pandas的value_counts()方法可以看loan_status這一列每種結果出現的次數的一個排名，我在最後只選擇了全額貸款（Fully Paid）以及不給放款（Charge Off）這兩個來作爲二分類的label值，分別用1和0表示

loans_2007 = loans_2007[(loans_2007['loan_status'] == "Fully Paid") | (loans_2007['loan_status'] == "Charged Off")]

status_replace = {
    "loan_status" : {
        "Fully Paid": 1,
        "Charged Off": 0,
    }
}
loans_2007 = loans_2007.replace(status_replace)

注意：剛剛處理完的數據還有32列

1.4去掉某一列的值是完全一樣的

orig_columns = loans_2007.columns
drop_columns = []
for col in orig_columns:
    col_series = loans_2007[col].dropna().unique() #需要先去掉缺失值，否則NAN也會被算一種，可能就不會出現某一列是全部一樣的
    if len(col_series) == 1:
        drop_columns.append(col)
loans_2007 = loans_2007.drop(drop_columns, axis=1)
print(drop_columns)
print(loans_2007.shape)
loans_2007.to_csv('filtered_loans_2007.csv', index=False)

1.5 缺失值處理

loans = pd.read_csv('filtered_loans_2007.csv')
null_counts = loans.isnull().sum()
print(null_counts)

處理方法：缺失值較少的去掉有缺失的樣本，缺失值多的直接去掉該列

loans = loans.drop("pub_rec_bankruptcies", axis=1)
loans = loans.dropna(axis=0)
print(loans.dtypes.value_counts())

需要將字符類型進行轉換

object_columns_df = loans.select_dtypes(include=["object"])#只選中了字符型的列
print(object_columns_df.iloc[0])

將字符型的數據今行處理，比如rent類型的替換成數值型

cols = ['home_ownership', 'verification_status', 'emp_length', 'term', 'addr_state']
for c in cols:
    print(loans[c].value_counts())

貸款的目的和標題可以只選擇其中之一，因爲title的可選項較多，所以選擇purpose

print(loans["purpose"].value_counts())
print(loans["title"].value_counts())

mapping_dict = {
    "emp_length": {
        "10+ years": 10,
        "9 years": 9,
        "8 years": 8,
        "7 years": 7,
        "6 years": 6,
        "5 years": 5,
        "4 years": 4,
        "3 years": 3,
        "2 years": 2,
        "1 year": 1,
        "< 1 year": 0,
        "n/a": 0
    }
}
loans = loans.drop(["last_credit_pull_d", "earliest_cr_line", "addr_state", "title"], axis=1)
loans["int_rate"] = loans["int_rate"].str.rstrip("%").astype("float")
loans["revol_util"] = loans["revol_util"].str.rstrip("%").astype("float")
loans = loans.replace(mapping_dict)

cat_columns = ["home_ownership", "verification_status", "emp_length", "purpose", "term"]
dummy_df = pd.get_dummies(loans[cat_columns])
loans = pd.concat([loans, dummy_df], axis=1)
loans = loans.drop(cat_columns, axis=1)
loans = loans.drop("pymnt_plan", axis=1)

loans.to_csv('cleaned_loans2007.csv', index=False)

2.網站獲取最大利潤的做法

loans = pd.read_csv("cleaned_loans2007.csv")
print(loans.info())

最大利潤情況：考慮光用精度可能不行，因爲樣本極其不平衡，借錢樣本很多，不借錢的樣本很少。要考慮FPR和TPR，也就是不還錢的人儘量不借，FPR越小，TPR越高越好。

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
lr = LogisticRegression(class_weight="balanced")
# kf = KFold(features.shape[0], random_state=1)
predictions = cross_val_predict(lr, features, target, cv=5)
predictions = pd.Series(predictions)

# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])

# Rates
tpr = tp / float((tp + fn))  #也可用roc_curve直接求
fpr = fp / float((fp + tn))

print(tpr)
print(fpr)
print predictions[:20]