Django Restfarmwork 查詢接口優化

1.儘量避免在for循環中進行數據庫查詢操作

相關代碼示例:

# 優化前寫法:
students = get_training_team_student_info(data.get('resource_id'))

for student in students:
    student_name = student.get('user_name')
    # 獲取參訓人員分數
    student_score_obj = get_training_team_score_first(
            training_team=data.get('resource_id'), 
            user=student.get('user_id')
            )
    if not student_score_obj:
        student_score = 0
    else:
        student_score = student_score_obj

# 優化後寫法:
students = get_training_team_student_info(data.get('resource_id'))
# 查出所有學員成績數據
student_score_lists = get_training_team_score(training_team=data['resource_id'])
student_score_dict = {}
for student_score in student_score_lists:
    student_score_dict[student_score.user_id] = student_score.final_score

for student in students:
    student_name = student.get('user_name')
    # 獲取參訓人員分數
    student_score_obj = student_score_dict.get(student.get('user_id'))

    if not student_score_obj:
        student_score = 0
    else:
        student_score = student_score_obj

描述: 根據實際情況先查出所有數據組成映射關係,在for循環中進行取值操作

2.對於使用 values_list,獲取一個字段結果的查詢方法,使用列表推導式生成一個字段的列表

相關代碼示例:

# 優化前相關代碼:
# 獲取角色下已下發的任務hash列表
send_missions = model_api.get_training_team_mission(
    training_team=training_team_hash,
    training_role=training_role.resource_id
).values_list('mission_resource_id', flat=True)

"""優化前SQL執行
此sql對應上面代碼的 send_missions = model_api.get_training_team_mission(training_team=training_team_hash,training_role=training_role.resource_id)
SELECT `ce_training_team_trainingteammission`.`id`, `ce_training_team_trainingteammission`.`resource_id`, `ce_training_team_trainingteammission`.`training_team`, `ce_training_team_trainingteammission`.`training_role`, `ce_training_team_trainingteammission`.`mission_resource_id`, `ce_training_team_trainingteammission`.`user_id`, `ce_training_team_trainingteammission`.`mission_status`, `ce_training_team_trainingteammission`.`user_answer`, `ce_training_team_trainingteammission`.`release_time`, `ce_training_team_trainingteammission`.`reporting_time`, `ce_training_team_trainingteammission`.`start_mission_time`, `ce_training_team_trainingteammission`.`status` FROM `ce_training_team_trainingteammission` WHERE (NOT (`ce_training_team_trainingteammission`.`status` = 0) AND `ce_training_team_trainingteammission`.`training_role` = 'ce66d38a3b0784ddd9074cecbc9bc288' AND `ce_training_team_trainingteammission`.`training_team` = 'ceb99b9d1677016bd09b38295422ee89'); args=(0, 'ce66d38a3b0784ddd9074cecbc9bc288', 'ceb99b9d1677016bd09b38295422ee89')

此sql對應上面代碼的.values_list('mission_resource_id', flat=True)
SELECT `ce_training_team_trainingteammission`.`mission_resource_id` FROM `ce_training_team_trainingteammission` WHERE (NOT (`ce_training_team_trainingteammission`.`status` = 0) AND `ce_training_team_trainingteammission`.`training_role` = 'ce66d38a3b0784ddd9074cecbc9bc288' AND `ce_training_team_trainingteammission`.`training_team` = 'ceb99b9d1677016bd09b38295422ee89')  LIMIT 21; args=(0, 'ce66d38a3b0784ddd9074cecbc9bc288', 'ceb99b9d1677016bd09b38295422ee89')

執行get_training_team_mission()函數時會進行一次數據庫查詢,執行.values_list時會再一次進行數據庫查詢
"""
# 優化後相關代碼:
send_missions = model_api.get_training_team_mission(
    training_team=training_team_hash,
    training_role=training_role.resource_id
)
mission_resource_id_list = [mission.mission_resource_id for mission in send_missions]
"""優化前SQL執行
SELECT `ce_training_team_trainingteammission`.`id`, `ce_training_team_trainingteammission`.`resource_id`, `ce_training_team_trainingteammission`.`training_team`, `ce_training_team_trainingteammission`.`training_role`, `ce_training_team_trainingteammission`.`mission_resource_id`, `ce_training_team_trainingteammission`.`user_id`, `ce_training_team_trainingteammission`.`mission_status`, `ce_training_team_trainingteammission`.`user_answer`, `ce_training_team_trainingteammission`.`release_time`, `ce_training_team_trainingteammission`.`reporting_time`, `ce_training_team_trainingteammission`.`start_mission_time`, `ce_training_team_trainingteammission`.`status` FROM `ce_training_team_trainingteammission` WHERE (NOT (`ce_training_team_trainingteammission`.`status` = 0) AND `ce_training_team_trainingteammission`.`training_role` = 'ce66d38a3b0784ddd9074cecbc9bc288' AND `ce_training_team_trainingteammission`.`training_team` = 'ceb99b9d1677016bd09b38295422ee89'); args=(0, 'ce66d38a3b0784ddd9074cecbc9bc288', 'ceb99b9d1677016bd09b38295422ee89')
 只會在執行get_training_team_mission()函數時時進行一次數據庫查詢
"""

3.在列表頁數據的獲取接口中,只返回列表頁需要數據,減少不必要數據的查詢

相關代碼示例:

# 優化前返回字段:
fields = ("resource_id", "name", "description", "category", "pre_skill", "train_skill", "start_time", "end_time",
            "template", "scene", "difficult", "scene_config_data", "training_role_data", "duration_time",
            "scene_config", "scene_start_time", "scene_end_time", "training_status", "scene_data", "roles_data")

# 優化後針對列表頁接口返回指定字段
def get_serializer(self, *args, **kwargs):
    """
    獲取序列化器

    :param args: 位置參數
    :param kwargs: 不定長字典參數
    :return: 序列化器
    """
    if self.action == 'list':
        kwargs['fields'] = (
            "resource_id", "name", "description", "category", "pre_skill", "train_skill", "start_time", "end_time",
            "template", "difficult", "scene_config", "training_role_data", "duration_time",
            "scene_start_time", "scene_end_time", "training_status"
        )
    return super().get_serializer(*args, **kwargs)

4.對於多對多的關聯查詢,使用 prefetch_related 進行預加載

相關代碼示例:

# 優化前相關代碼:
role_list = get_training_role(
        training_team=data['resource_id']
    ).exclude(users__id=user_id)

for role in role_list:
    role_name = role.name
    users = role.users.all()
    user_names = [user.username for user in users]
    role_info_list.append({
        'role_name': role_name,
        'users': user_names
    })
"""優化前執行sql
此sql對應上面代碼的 role_list = get_training_role(training_team=data['resource_id'])
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE `ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54'; args=('a0b8cda8d219f5a4af005e73c542ef54',)

此sql對應上面代碼的 .exclude(users__id=user_id)
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE (`ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54' AND NOT (`ce_training_team_trainingrole`.`id` IN (SELECT U1.`trainingrole_id` FROM `ce_training_team_trainingrole_users` U1 WHERE U1.`user_id` = 7))); args=('a0b8cda8d219f5a4af005e73c542ef54', 7)

此sql對應上面代碼的 users = role.users.all()
SELECT `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` = 235; args=(235,)
SELECT `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` = 236; args=(236,)
SELECT `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` = 237; args
"""

# 優化後相關代碼
role_list = model_api.get_training_role(
        training_team=data['resource_id']
    ).exclude(users__id=user_id).prefetch_related('users')

for role in role_list:
    role_name = role.name
    users = role.users.all()
    user_names = [user.username for user in users]
    role_info_list.append({
        'role_name': role_name,
        'users': user_names
    })

"""優化後執行sql
此sql對應上面代碼的 role_list = get_training_role(training_team=data['resource_id'])
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE `ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54'; args=('a0b8cda8d219f5a4af005e73c542ef54',)

此sql對應上面代碼的 .exclude(users__id=user_id)
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE (`ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54' AND NOT (`ce_training_team_trainingrole`.`id` IN (SELECT U1.`trainingrole_id` FROM `ce_training_team_trainingrole_users` U1 WHERE U1.`user_id` = 7))); args=('a0b8cda8d219f5a4af005e73c542ef54', 7)

此sql對應上面代碼的 .prefetch_related('users')
SELECT (`ce_training_team_trainingrole_users`.`trainingrole_id`) AS `_prefetch_related_val_trainingrole_id`, `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` IN (235, 236, 237); args=(235, 236, 237)

相對來說此例子只是少了一個sql,但是如果這個角色下的人員有很多的話,每一個人都會進行一次sql查詢,
而優化後不管有多少人,只是進行了一次sql查詢
"""

拓展: 相關select_related()和prefetch_related()的操作可參考:https://www.cnblogs.com/tuifeideyouran/p/4232028.html

5.儘可能少的在緩存函數後面添加其他操作,例如.first(),.exclude(),.values_list(),.values()等相關操作

相關代碼示例:

# 優化後相關代碼
@func_cache(ce_training_team_teacher_cache)
def get_training_role(**kwargs):
    """
    獲取團隊訓練實戰角色
    :param kwargs:參數
    :return: 團隊訓練實戰角色obj
    """
    return TrainingRole.objects.filter(**kwargs)


@func_cache(ce_training_team_teacher_cache)
def get_training_role_first(**kwargs):
    """
    獲取團隊訓練實戰角色
    :param kwargs:參數
    :return: 團隊訓練實戰角色obj
    """
    return TrainingRole.objects.filter(**kwargs).first()


@func_cache(ce_training_team_teacher_cache)
def get_training_role_prefetch(**kwargs):
    """
    獲取團隊訓練實戰角色預加載角色信息
    :param kwargs: 參數
    :return: 團隊訓練實戰角色obj
    """
    return TrainingRole.objects.filter(**kwargs).prefetch_related('users')

儘可能通過python的邏輯關係在查詢數據庫之前排除不必要的條件,例如之前的values_list()獲取單個字段數據時,可使用for循環代替.
對於使用較多的情況,可針對條件設置單個緩存函數,例如上面的get_training_role_prefetch()函數.
每添加一個額外操作都會增加一次sql查詢,針對這個特性,具體要根據實際的代碼進行調整