linux tc

Linux TC 命令

Traffic Control 流量控制

Posted by LXG on November 7, 2019

深入理解android-看云

InterfaceCmd-看云

Linux TC-gitub

概念

TC 是 Traffic Control 的缩写。在Linux中，流量控制是通过建立数据包队列(Queue)，并控制各个队列中数据包的发送方式实现的

基本工作原理

linux_tc

  • 接收包从输入接口 (Input Interface) 进来后，将经过输入流量限制 (Ingress Policing) 以丢弃不符合规定的数据包。而符合规定的数据包则交给输入多路选择器材 (Input De-Multiplexing) 进行判断选择

  • 输入多路选择器的选择结果是，如果数据包的目地是本机，则将该包送给上层处理，否则需要将数据交到转发块 (Forwarding Blcok) 去处理。转发块同时也接收来自本机上层 (TCP\UDP) 产生的数据包

  • 转发块通过查看路由表，决定处理包的下一个目的地。然后，转发块对数据包进行排列整合以便将它们送到对应的输出接口 (Output Interface)

具体实现

系统会建立许多队列及对应的队列规则 (queuing disipline, 简称qdisc)

  • 针对网络物理设备（如以太网卡eth0）绑定一个队列qdisc

  • 在该队列上建立class

  • 为每一个分类建立一个基于路由的过滤器filter

  • 最后与过滤器配合，建立特定的路由表

无分类的队列规则（Classless qdisc）

该规则对进入网卡的数据包不加区分，统一对待。使用这种规定的处理能否对数据包重新编排、延迟或丢弃。这种类型是针对整个网卡的流量进行调整。

常用的qdisc如下：

  • fifo(First In First Out, 先进先出队列)：最简单的控制

  • SFQ(Stochastic Fairness Queuing, 随机公平队列)：对发送会话进行重排，每个发送会话可以公平的发送数据

  • RED(Random Eraly Detection, 前向随机丢包)：用于模拟流量接近带宽限制时丢包的情况

  • TBF(Token Bucket Fliter, 令牌桶过滤器)：可较好地使得流量降低到预设值。适合高带宽的环境。这类qdisc使用的流量控制手段主要是排序、限速、丢包

InterfaceCmd

InterfaceCmd 用来管理和控制系统中的网络设备


system/netd

./server/RouteController.cpp
./server/BandwidthController.cpp
./server/ResolverController.cpp
./server/PPPOEController.cpp
./server/TetherController.cpp
./server/StrictController.cpp
./server/IPv6TetherController.cpp
./server/FirewallController.cpp
./server/PerfController.cpp
./server/NetworkController.cpp
./server/PppController.cpp
./server/IdletimerController.cpp
./server/ClatdController.cpp
./server/SoftapController.cpp
./server/ThrottleController.cpp
./server/InterfaceController.cpp
./server/BridgeController.cpp
./server/Controllers.cpp
./server/NatController.cpp

背景知识

虚拟网络设备

Linux虚拟网络-IBM

IFB设备

IFB (Intermediate Functional Block) 是 IMQ (Intermediate Queuing) 的替代者，二者都是Linux为更好地完成流量控制而实现的虚拟设备。

Linux 丰富的流量控制手段和规则都是针对出口流量的，即大多数排队规则（qdisc）都是用于输出方向的。而输入方向主要是入口流量控制只有一个规则，即ingress qdisc, 系统新增了一种方式，用于重定向incoming packets，通过ingress disc 把输入方向的数据包重定向到虚拟设备IFB，而IFB的输出方向配置多种qdisc, 就可以达到对输入方向的流量做队列调度的目的

linux_tc_ifb

网络设备编程

Netdevice是Linux平台中直接针对底层网络设备编程的一套接口，其使用方法很简单，就是利用socket句柄和ioctl函数来操作指定的网络设备

system/core/libnetutils/ifc_utils.c


static void ifc_init_ifr(const char *name, struct ifreq *ifr)
{
    memset(ifr, 0, sizeof(struct ifreq));
    strncpy(ifr->ifr_name, name, IFNAMSIZ);
    ifr->ifr_name[IFNAMSIZ - 1] = 0;
}

static int ifc_set_flags(const char *name, unsigned set, unsigned clr)
{
    struct ifreq ifr;
    ifc_init_ifr(name, &ifr);

    if(ioctl(ifc_ctl_sock, SIOCGIFFLAGS, &ifr) < 0) return -1;
    ifr.ifr_flags = (ifr.ifr_flags & (~clr)) | set;
    return ioctl(ifc_ctl_sock, SIOCSIFFLAGS, &ifr);
}

int ifc_up(const char *name)
{
    int ret = ifc_set_flags(name, IFF_UP, 0);
    if (DBG) printerr("ifc_up(%s) = %d", name, ret);
    return ret;
}

int ifc_down(const char *name)
{
    int ret = ifc_set_flags(name, 0, IFF_UP);
    if (DBG) printerr("ifc_down(%s) = %d", name, ret);
    return ret;
}

代码

system/netd/server/ThrottleController.cpp


#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <string.h>

#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <linux/pkt_sched.h>

#define LOG_TAG "ThrottleController"
#include <cutils/log.h>
#include <netutils/ifc.h>

#include "ThrottleController.h"
#include "NetdConstants.h"

extern "C" int ifc_init(void);
extern "C" int ifc_up(const char *name);
extern "C" int ifc_down(const char *name);

ThrottleController::ThrottleController(){
    mModemRx = -1;
    mModemTx = -1;
}
ThrottleController::~ThrottleController(){
}

int ThrottleController::runTcCmd(const char *cmd) {

    ALOGE("runTcCmd: %s", cmd);

    char *buffer;
    size_t len = strnlen(cmd, 255);
    int res;

    if (len == 255) {
        ALOGE("tc command too long");
        errno = E2BIG;
        return -1;
    }

    asprintf(&buffer, "%s %s", TC_PATH, cmd);
    res = system_nosh(buffer);
    free(buffer);
    return res;
}

int ThrottleController::setInterfaceThrottle(const char *iface, int rxKbps, int txKbps) {
    char cmd[512];
    char ifn[65];

    memset(ifn, 0, sizeof(ifn));
    strncpy(ifn, iface, sizeof(ifn)-1);

    if (txKbps == -1) {
        reset(ifn);
        return 0;
    }

    /*
     * by mtk80842, reset configuration before setting
     */
    reset(ifn);

    /*
     *
     * Target interface configuration
     *
     */

    /*
     * Add root qdisc for the interface
     */
    sprintf(cmd, "qdisc add dev %s root handle 1: htb default 1 r2q 1000", ifn);
    if (runTcCmd(cmd)) {
        ALOGE("Failed to add root qdisc (%s)", strerror(errno));
        goto fail;
    }

    /*
     * Add our egress throttling class
     */
    sprintf(cmd, "class add dev %s parent 1: classid 1:1 htb rate %dkbit", ifn, txKbps);
    if (runTcCmd(cmd)) {
        ALOGE("Failed to add egress throttling class (%s)", strerror(errno));
        goto fail;
    }

    if(rxKbps == -1){
        ALOGI("setInterfaceThrottle success but NO RX, ifn = %s", ifn);
        return 0;
    }

    /*
     * Bring up the IFD device
     */
    ifc_init();
    if (ifc_up("ifb0")) {
        ALOGE("Failed to up ifb0 (%s)", strerror(errno));
        goto fail;
    }

    /*
     * Add root qdisc for IFD
     */
    sprintf(cmd, "qdisc add dev ifb0 root handle 1: htb default 1 r2q 1000");
    if (runTcCmd(cmd)) {
        ALOGE("Failed to add root ifb qdisc (%s)", strerror(errno));
        goto fail;
    }

    /*
     * Add our ingress throttling class
     */
    sprintf(cmd, "class add dev ifb0 parent 1: classid 1:1 htb rate %dkbit", rxKbps);
    if (runTcCmd(cmd)) {
        ALOGE("Failed to add ingress throttling class (%s)", strerror(errno));
        goto fail;
    }

    /*
     * Add ingress qdisc for pkt redirection
     */
    sprintf(cmd, "qdisc add dev %s ingress", ifn);
    if (runTcCmd(cmd)) {
        ALOGE("Failed to add ingress qdisc (%s)", strerror(errno));
        goto fail;
    }

    /*
     * Add filter to link <ifn> -> ifb0
     */
    sprintf(cmd, "filter add dev %s parent ffff: protocol ip prio 10 u32 match "
            "u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0", ifn);
    if (runTcCmd(cmd)) {
        ALOGE("Failed to add ifb filter (%s)", strerror(errno));
        goto fail;
    }

    ALOGI("setInterfaceThrottle success, ifn = %s", ifn);

    return 0;
fail:
    reset(ifn);
    return -1;
}

void ThrottleController::reset(const char *iface) {
    char cmd[128];

    ALOGI("reset %s qdisc", iface);

    sprintf(cmd, "qdisc del dev %s root", iface);
    runTcCmd(cmd);
    sprintf(cmd, "qdisc del dev %s ingress", iface);
    runTcCmd(cmd);

    runTcCmd("qdisc del dev ifb0 root");
}

int ThrottleController::getInterfaceRxThrottle(const char *iface, int *rx) {
    ALOGI("getInterfaceRxThrottle %s", iface);
    *rx = 0;
    return 0;
}

int ThrottleController::getInterfaceTxThrottle(const char *iface, int *tx) {
    ALOGI("getInterfaceTxThrottle %s", iface);
    *tx = 0;
    return 0;
}

int ThrottleController::getModemRxThrottle(int *rx) {
    *rx = mModemRx;
    return 0;
}

int ThrottleController::getModemTxThrottle(int *tx) {
    *tx = mModemTx;
    return 0;
}

int ThrottleController::setModemThrottle(int rxKbps, int txKbps) {
    unsigned int n = 0; unsigned int mask = 0;
    int idx = 0; unsigned int up = 0;
    // qcom: "rmnet_data0"
    const char* iface_list[] = {
      "ccmni0",
      "ccmni1",
      "ccmni2",
      0
    };

    mModemRx = rxKbps;
    mModemTx = txKbps;
    ifc_init();
    while(iface_list[idx] != 0){
        up = 0;
        ifc_is_up(iface_list[idx], &up);
        if(up == 1){
            n++;
            mask |= (1 << idx);
            ALOGD("%s is up, n = %d, mask = 0x%x", iface_list[idx], n, mask);
        }
        idx++;
    }
    ifc_close();
    if(n == 0){
        ALOGE("setModemThrottle: no modem interface is up !");
        return -1;
    }
    if(rxKbps >= 0)
        rxKbps = rxKbps/n;
    if(txKbps >= 0)
        txKbps = txKbps/n;

    idx = 0;
    while(iface_list[idx] != 0){
        if(mask & (1 << idx)){
            ALOGD("setModemThrottle for %s: rx %d, tx %d !",
            iface_list[idx], rxKbps, txKbps);
            setInterfaceThrottle(iface_list[idx], rxKbps, txKbps);
        }
        idx++;
    }

    return 0;
}

MTK 日志


11-07 18:07:24.613   446  1084 D FrameworkListener: dispatchCommand data = (103 interface setthrottle modem 80 160)
11-07 18:07:24.613   446  1084 D ThrottleController: ccmni1 is up, n = 1, mask = 0x2
11-07 18:07:24.613   446  1084 D ThrottleController: setModemThrottle for ccmni1: rx 80, tx 160 !
11-07 18:07:24.613   446  1084 I ThrottleController: reset ccmni1 qdisc
11-07 18:07:24.613   446  1084 E ThrottleController: runTcCmd: qdisc del dev ccmni1 root
11-07 18:07:24.651   446  1084 E ThrottleController: runTcCmd: qdisc del dev ccmni1 ingress
11-07 18:07:24.695   446  1084 E ThrottleController: runTcCmd: qdisc del dev ifb0 root
11-07 18:07:24.734   446  1084 E ThrottleController: runTcCmd: qdisc add dev ccmni1 root handle 1: htb default 1 r2q 1000
11-07 18:07:24.768   446  1084 E ThrottleController: runTcCmd: class add dev ccmni1 parent 1: classid 1:1 htb rate 160kbit
11-07 18:07:24.801   446  1084 D DHCP    : ifc_up(ifb0) = 0
11-07 18:07:24.801   446  1084 E ThrottleController: runTcCmd: qdisc add dev ifb0 root handle 1: htb default 1 r2q 1000
11-07 18:07:24.834   446  1084 E ThrottleController: runTcCmd: class add dev ifb0 parent 1: classid 1:1 htb rate 80kbit
11-07 18:07:24.868   446  1084 E ThrottleController: runTcCmd: qdisc add dev ccmni1 ingress
11-07 18:07:24.901   446  1084 E ThrottleController: runTcCmd: filter add dev ccmni1 parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress
redirect dev ifb0
11-07 18:07:24.934   446  1084 I ThrottleController: setInterfaceThrottle success, ifn = ccmni1

QCOM 日志


11-07 18:25:52.721   769  1538 D ThrottleController: rmnet_data0 is up, n = 1, mask = 0x1
11-07 18:25:52.721   769  1538 D ThrottleController: setModemThrottle for rmnet_data0: rx 80, tx 80 !
11-07 18:25:52.721   769  1538 I ThrottleController: reset rmnet_data0 qdisc
11-07 18:25:52.721   769  1538 E ThrottleController: runTcCmd: qdisc del dev rmnet_data0 root
11-07 18:25:52.802   769  1538 E ThrottleController: runTcCmd: qdisc del dev rmnet_data0 ingress
11-07 18:25:52.895   769  1538 E ThrottleController: runTcCmd: qdisc del dev ifb0 root
11-07 18:25:52.946   769  1538 E ThrottleController: runTcCmd: qdisc add dev rmnet_data0 root handle 1: htb default 1 r2q 1000
11-07 18:25:52.999   769  1538 E ThrottleController: runTcCmd: class add dev rmnet_data0 parent 1: classid 1:1 htb rate 80kbit
11-07 18:25:53.057   769  1538 E ThrottleController: runTcCmd: qdisc add dev ifb0 root handle 1: htb default 1 r2q 1000
11-07 18:25:53.109   769  1538 E ThrottleController: runTcCmd: class add dev ifb0 parent 1: classid 1:1 htb rate 80kbit

11-07 18:25:53.164   769  1538 E ThrottleController: runTcCmd: qdisc add dev rmnet_data0 ingress
11-07 18:25:53.228   769  1538 E ThrottleController: runTcCmd: filter add dev rmnet_data0 parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0

// fail
11-07 18:25:53.317   769  1538 E ThrottleController: Failed to add ifb filter (No such device)
11-07 18:25:53.317   769  1538 I ThrottleController: reset rmnet_data0 qdisc
11-07 18:25:53.317   769  1538 E ThrottleController: runTcCmd: qdisc del dev rmnet_data0 root
11-07 18:25:53.375   769  1538 E ThrottleController: runTcCmd: qdisc del dev rmnet_data0 ingress
11-07 18:25:53.427   769  1538 E ThrottleController: runTcCmd: qdisc del dev ifb0 root
CATALOG
    FEATURED TAGS
    Android debug dumpsys shell linux Permission 心理学 ubuntu JNI markdown android IMS systemui selinux eSIM androidstudio performance pms wms view flutter iptables netd property c++ kernel mqtt usb company statusbar docker hardware webrtc recovery web camera rk3288 A133 driver rockchip rk3399 chip rk3588 a133 qt t113 buildroot AI MCU
    FRIENDS