ReYOLOv8流水笔记
一直想找一个带LSTM的目标检测神经网络，问了AI以为ReYOLOv8是，结果发现跟我预期的不一致，不过既然调试了，就先记录一下，主要关注神经网络部分
原项目地址为 https://github.com/silvada95/ReYOLOv8
我写不出这么庞大复杂的代码，我把代码下载下来运行，发现运行不太正常，当然我也没有调试，而是直接扔给了AI，AI帮我修复了一下，我的项目地址是 https://github.com/moneypi/ReYOLOv8，我只调试了val.py
调试打印出来的核心的神经网络结构如下
AutoBackendMemory(
(model): DetectionModel2(
(model): Sequential(
(0): Conv(
(conv): Conv2d(5, 24, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(24, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): C2f(
(cv1): Conv(
(conv): Conv2d(48, 48, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(96, 48, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(24, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(24, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(24, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(24, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(3): Conv_LSTM(
(default_act): Tanh()
(default_act2): Sigmoid()
(Gates): Conv2d(96, 192, kernel_size=(1, 1), stride=(1, 1))
)
(4): Conv(
(conv): Conv2d(48, 88, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(5): C2f(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(220, 88, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(cv1): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(6): Conv_LSTM(
(default_act): Tanh()
(default_act2): Sigmoid()
(Gates): Conv2d(176, 352, kernel_size=(1, 1), stride=(1, 1))
)
(7): Conv(
(conv): Conv2d(88, 176, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(8): C2f(
(cv1): Conv(
(conv): Conv2d(176, 176, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(440, 176, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(9): Conv_LSTM(
(default_act): Tanh()
(default_act2): Sigmoid()
(Gates): Conv2d(352, 704, kernel_size=(1, 1), stride=(1, 1))
)
(10): Conv(
(conv): Conv2d(176, 344, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(11): C2f(
(cv1): Conv(
(conv): Conv2d(344, 344, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(688, 344, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(12): Conv_LSTM(
(default_act): Tanh()
(default_act2): Sigmoid()
(Gates): Conv2d(688, 1376, kernel_size=(1, 1), stride=(1, 1))
)
(13): SPPF(
(cv1): Conv(
(conv): Conv2d(344, 172, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(688, 344, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): MaxPool2d(kernel_size=5, stride=1, padding=2, dilation=1, ceil_mode=False)
)
(14): Upsample(scale_factor=2.0, mode=nearest)
(15): Concat()
(16): C2f(
(cv1): Conv(
(conv): Conv2d(520, 176, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(352, 176, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(17): Upsample(scale_factor=2.0, mode=nearest)
(18): Concat()
(19): C2f(
(cv1): Conv(
(conv): Conv2d(264, 88, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(176, 88, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(20): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(21): Concat()
(22): C2f(
(cv1): Conv(
(conv): Conv2d(264, 176, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(352, 176, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(23): Conv(
(conv): Conv2d(176, 176, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(act): SiLU(inplace=True)
)
(24): Concat()
(25): C2f(
(cv1): Conv(
(conv): Conv2d(520, 344, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(688, 344, kernel_size=(1, 1), stride=(1, 1))
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): Bottleneck(
(cv1): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(cv1): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(cv2): Conv(
(conv): Conv2d(172, 172, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
)
)
)
(26): Detect(
(cv2): ModuleList(
(0): Sequential(
(0): Conv(
(conv): Conv2d(88, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1))
)
(1): Sequential(
(0): Conv(
(conv): Conv2d(176, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1))
)
(2): Sequential(
(0): Conv(
(conv): Conv2d(344, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1))
)
)
(cv3): ModuleList(
(0): Sequential(
(0): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): Conv2d(88, 2, kernel_size=(1, 1), stride=(1, 1))
)
(1): Sequential(
(0): Conv(
(conv): Conv2d(176, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): Conv2d(88, 2, kernel_size=(1, 1), stride=(1, 1))
)
(2): Sequential(
(0): Conv(
(conv): Conv2d(344, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(1): Conv(
(conv): Conv2d(88, 88, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): SiLU(inplace=True)
)
(2): Conv2d(88, 2, kernel_size=(1, 1), stride=(1, 1))
)
)
(dfl): DFL(
(conv): Conv2d(16, 1, kernel_size=(1, 1), stride=(1, 1), bias=False)
)
)
)
)
)debug的时候主要是关注 ReYOLOv8/ultralytics/nn/tasks.py 文件，DetectionModel2 的 forward 调用的 _forward_once，输入是一个 (1, 5, 32, 32) 的数据，所以我对这个神经网络不满意的地方就在这里，他是针对特定的摄像机的数据进行推理的，而不是RGB，或者哪怕是YUV, Raw Bayer(12-16bit) 这种数据，而是如其标题所说，是 Event-Based，无法像Yolov8一样推理通用的输入数据
先是执行了几个卷积神经网络，尺寸变为了 (1, 48, 8, 8)
然后跟了 (3) Conv_LSTM，输出是两个(1, 48, 8, 8)，保存在 hidden_states["0"] 中，hidden_states = {"0": None, "1": None, "2": None, "3": None}
然后x = hidden_states["0"][0]，又执行了几个卷积，尺寸变为了 (1, 88, 4, 4)
然后跟 (6) Conv_LSTM，输出是两个 (1, 88, 4, 4), 保存在 hidden_states["1"] 中，
然后x = hidden_states["1"][0]，又执行了几个卷积，尺寸变为了 (1, 176, 2, 2)
然后跟 (9) Conv_LSTM，输出是两个 (1, 176, 2, 2), 保存在 hidden_states["2"] 中，
然后x = hidden_states["2"][0]，又执行了几个卷积，尺寸变为了 (1, 344, 1, 1)
然后跟 (12) Conv_LSTM，输出是两个 (1, 344, 1, 1), 保存在 hidden_states["3"] 中，
然后x = hidden_states["3"][0]，又执行了几个卷积，又加了上采样，尺寸变为了 (1, 344, 2, 2)
看起来是从y中获取数据，x变为了len 为2的list，[0]为 (1, 344, 2, 2)，[1]为 (1, 176, 2, 2)
(15) concat后变为了(1, 520, 2, 2)
经过 (16): C2f 之后变为了 (1, 176, 2, 2)，经过 (17): Upsample 变为了 (1, 176, 4, 4)
(18): Concat是吧x 和 y[6] 拼接， y[6]尺寸为 (1, 88, 4, 4)，拼接后是 [1, 264, 4, 4]
经过 (19): C2f 和 (20): Conv 后尺寸变为 (1, 88, 2, 2)
(21): Concat 是拼接 y[16]， 尺寸为 [1, 176, 2, 2]，拼接后是 [1, 264, 2, 2]
经过 (22): C2f 和 (23): Conv 后，尺寸变为 [1, 176, 1, 1]
(24): Concat 是拼接 y[13]，尺寸为 (1, 344, 1, 1)，拼接后为 (1, 520, 1, 1)
经过(25): C2f后，尺寸变为 (1, 344, 1, 1)
x是y[19, 22, 25]的拼接，就是 (1, 88, 44) (1, 176, 2, 2) (1, 344, 1, 1)
接(26): Detect后，生成的x[0].shape = [1, 6, 21],
x[1][0].shape=[1, 66, 4, 4]
x[1][1].shape=[1, 66, 2, 2]
x[1][2].shape=[1, 66, 1, 1]
然后返回了 x 和 hidden_states "0"是(1, 48, 8, 8)，"1"是(1, 88, 4, 4)，"2"是(1, 176, 2, 2)，"3"是 (1, 344, 1, 1)， "4"是 None
y会保存[6, 9, 13, 16, 19, 22, 25]层输出
调试出x是(1, 5, 256, 320), 经过(0): Conv，(1): Conv和 (2): C2f后，x尺寸变为 torch.Size([1, 48, 64, 80]), y[0]y[1]y[2]均为None
经过(3): Conv_LSTM之后，生成了两个 torch.Size([1, 48, 64, 80])，保存到hidden_states['0']，其中[0]赋值给x, y[3]为None
经过(4): Conv和(5): C2f后，x.shape=torch.Size([1, 88, 32, 40]),y[4]和y[5]均为None
(6): Conv_LSTM生成两个，torch.Size([1, 88, 32, 40])，保存到hidden_states['1']，其中[0]赋值给[0]，并保存为y[6]
经过(7): Conv和(8): C2f之后，x.shape=torch.Size([1, 176, 16, 20]),y[7]和y[8]均为None
(9): Conv_LSTM生成了两个torch.Size([1, 176, 16, 20]),保存到hidden_states['2']，其中[0]赋值给[0]，并保存为y[9]
经过(10): Conv和(11): C2f之后，x.shape=torch.Size([1, 344, 8, 10])，y[10]和y[11]均为None
(12): Conv_LSTM生成两个 torch.Size([1, 344, 8, 10]), 保存到hidden_states['3']，其中[0]赋值给[0]，并保存为y[12]为None
(13): SPPF后x.shape=torch.Size([1, 344, 8, 10])并保存到y[13]
(14): Upsample后x.shape=torch.Size([1, 344, 16, 20]),y[14]为None
(15): Concat是拼接x和y[9],拼接为torch.Size([1, 520, 16, 20])，y[15]为None
(16): C2f后x.shape=torch.Size([1, 176, 16, 20])，保存为y[16]
(17): Upsample后是x.shape=torch.Size([1, 176, 32, 40])，y[17]为None
(18): Concat是拼接y[6],得到 x.shape=torch.Size([1, 264, 32, 40]),y[18]为None
(19): C2f后x.shape=torch.Size([1, 88, 32, 40]),保存到y[19]
(20): Conv后x.shape=torch.Size([1, 88, 16, 20]),y[20]为None
(21): Concat是拼接 y[16]，得到 x.shape=torch.Size([1, 264, 16, 20]),y[21]为None
(22): C2f后得到x.shape=torch.Size([1, 176, 16, 20])，并保存到y[22]
(23): Conv后得到 x.shape=torch.Size([1, 176, 8, 10])，y[23]为None
(24): Concat是拼接y[13]得到x.shape=torch.Size([1, 520, 8, 10]),y[24]为None
(25): C2f得到x.shape=torch.Size([1, 344, 8, 10])，保存到y[25]
(26): Detect先是拼接了y[19, 22, 25]，得到x[0].shape=torch.Size([1, 88, 32, 40])
x[1].shape=torch.Size([1, 176, 16, 20])
x[2].shape=torch.Size([1, 344, 8, 10])
然后执行模型推理，代码在ReYOLOv8/ultralytics/nn/modules.py，Detect的forward函数，
x[0]需要cv2[0]和cv3[0]的推理并concat，得到x[0].shape=torch.Size([1, 66, 32, 40])
x[1]需要cv2[1]和cv3[1]的推理并concat，得到x[1].shape=torch.Size([1, 66, 16, 20])
x[2]需要cv2[2]和cv3[2]的推理并concat，得到x[2].shape=torch.Size([1, 66, 8, 10])
得到x[0].shape=torch.Size([1, 6, 1680])，就是box和cls，而上面的x[0],x[1],x[2]作为一个整体保存到x[1]里
Agent 垂直技术社区，欢迎活跃、内容共建。
更多推荐
7
0- 0
已为社区贡献3条内容
所有评论(0)