【有奖征文】EdgeX + OpenVINO 实现边缘智能 AI 应用

架构设计图本系统设计硬件设计采用 Intel® CPU(iGPU) + GPU[可选] ，CPU 必须为 Intel® 架构，独立 GPU 可根据实际需要灵活扩展。本系统全部采用 Docker 微服务运行，描述如下：图例中，①作为 AI 推理服务器，运行 OpenVINO™ Model Server 容

亿琪软件

206人浏览 · 2024-07-23 12:57:13

亿琪软件 · 2024-07-23 12:57:13 发布

架构设计图

本系统设计硬件设计采用 Intel® CPU(iGPU) + GPU[可选] ，CPU 必须为 Intel® 架构，独立 GPU 可根据实际需要灵活扩展。

本系统全部采用 Docker 微服务运行，描述如下：

图例中，① 作为 AI 推理服务器，运行 OpenVINO™ Model Server 容器；
图例中，② 作为流媒体服务器，负责流媒体的编解码，实时查看视频流，运行亿琪软件产品 YiMEDIA，同样也采用容器运行；
图例中，③ 作为边缘计算，运行 EdgeXFoundry 容器，负责整个系统的协调和业务驱动；

应用配置

硬件环境

软件环境

采用 Ubuntu 22.04 作为 Docker 宿主主机，并且已经成功完成 Docker 运行环境的安装和测试。

# uname -a
Linux YiFUSION-N100 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb  9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy

采用 CPU 内置 iGPU 作为推理

# ll /dev/dri/
total 0
drwxr-xr-x   3 root root        100  2月 23 17:34 ./
drwxr-xr-x  20 root root       4620  2月 23 20:19 ../
crw-rw----+  1 root video  226,   0  2月 23 20:58 card0
crw-rw----+  1 root render 226, 128  2月 23 17:39 renderD128

OpenVINO Model Server 服务

模型库配置：

name: 模型名称
base_path: 模型文件路径
target_device: 目标推理设备
layout: 需要的图片格式(可选)

# more models/config.json
{
  "model_config_list":[
      {
        "config":{
            "name":"ssdlite_mobilenet_v2",
            "base_path":"/models/ssdlite_mobilenet_v2",
            "target_device": "GPU"
        }
      },
  ]
}

既可以用云存储作为模型库存放位置，也可以使用本地磁盘作为模型库存放位置（本例子使用）。

一级目录是对应的模型名称
二级目录是模型的版本，比如：1，2...
xml 是 OpenVINO™ 模型库相关配置，bin 是 OpenVINO™ 模型 IR 文件

models 目录结构如下：

models/
├── ssdlite_mobilenet_v2
│   └── 1
│       ├── coco_91cl_bkgr.txt
│       ├── ssdlite_mobilenet_v2.bin
│       ├── ssdlite_mobilenet_v2.mapping
│       └── ssdlite_mobilenet_v2.xml

启动 Model Server 容器

docker run -d \
--name model_server \
--network edgex_edgex-network \
--device /dev/dri \
-v $(pwd)/models:/models \
-p 9000:9000 \
-p 8000:8000 \
openvino/model_server:latest-gpu \
--config_path /models/config.json \
--port 9000 \
--rest_port 8000 \
--log_level DEBUG

通过 Kserve RESTful API 访问 OVMS 工作状态，这里访问已经启动的模型：ssdlite_mobilenet_v2。

curl http://192.168.198.8:8000/v1/config
{
  "ssdlite_mobilenet_v2": {
    "model_version_status": [
      {
        "version": "1",
        "state": "AVAILABLE",
        "status": {
          "error_code": "OK",
          "error_message": "OK"
        }
      }
    ]
  }
}

同样通过 Kserve RESTful API 可以访问模型的详细输入和输出参数，主要是 inputs 和 outputs 里面的 dim。

inputs: 需要输入图片名称是 image_tensor，格式是 DT_INT8，可以用 byte。图片纬度是 [1,300,300,3]， width:300, height:300, color:3；
outputs: 输出格式是一个四维矩阵 [1,1,100,7]，只需要后面两个维度：rows:100, cols:7；

{
  "modelSpec": {
    "name": "ssdlite_mobilenet_v2",
    "signatureName": "",
    "version": "1"
  },
  "metadata": {
    "signature_def": {
      "@type": "type.googleapis.com/tensorflow.serving.SignatureDefMap",
      "signatureDef": {
        "serving_default": {
          "inputs": {
            "image_tensor": {
              "dtype": "DT_UINT8",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "300",
                    "name": ""
                  },
                  {
                    "size": "300",
                    "name": ""
                  },
                  {
                    "size": "3",
                    "name": ""
                  }
                ],
                "unknownRank": false
              },
              "name": "image_tensor"
            }
          },
          "outputs": {
            "detection_boxes": {
              "dtype": "DT_FLOAT",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "100",
                    "name": ""
                  },
                  {
                    "size": "7",
                    "name": ""
                  }
                ],
                "unknownRank": false
              },
              "name": "detection_boxes"
            }
          },
          "methodName": "",
          "defaults": {

          }
        }
      }
    }
  }
}

笔者采用亿琪软件公司自己的流媒体服务器软件：YiMEDIA，来实现视频流的编解码。

当然，用户也可以使用一些其他框架来支撑，比如：

EdgeX Foundry 服务

根据 EdgeXFoundry edgex-compose 仓库手册，创建自己的 EdgeX 运行环境。本测试案例，只需要核心服务和 mqtt-broker 服务即可。

Device Service for ONVIF Camera 是否需要使用，取决于是否由 EdgeX 来管理网络摄像头(IPC)，本例中暂时未启用此服务，采用手动设置。

这是一个系列微服务，包含了各种算法库，支持算法调用和推理。

SSDLITE_MOBILENET_V2：提交一张图片，返回目标识别结果；

deviceList:

- name: OpenVINO-Object-Detection
  profileName: OpenVINO-Device
  description: Example of OpenVINO Device
  labels: [AI]
  protocols:
    ovms:
      Host: 192.168.198.8
      Port: 9000
      Model: ssdlite_mobilenet_v2
      Version: 1
      Uri: rtsp://192.168.123.12:18554/test
      Snapshot: false
      Record: false
      Score: 0.3

这是一个设备服务基础配置例子，用户可根据实际业务需求配置：

Host: OVMS 服务地址
Port: OVMS 服务端口
Model: 模型名称
Version: 模型版本
Uri: IPC 摄像头流地址
Snapshot: 是否建立快照
Record: 是否记录推理视频
Score: 推理结果最低信任度

以下是完整的 EdgeX metadata：

{
  "apiVersion": "v3",
  "statusCode": 200,
  "totalCount": 1,
  "devices": [
    {
      "created": 1708691437289,
      "modified": 1708693207326,
      "id": "6b02c540-eeae-4b61-b741-cdefcf93fd09",
      "name": "OpenVINO-Object-Detection",
      "description": "Example of OpenVINO Device",
      "adminState": "UNLOCKED",
      "operatingState": "UP",
      "labels": ["AI"],
      "serviceName": "device-openvino-ssdlite-mobilenet-v2",
      "profileName": "OpenVINO-Device",
      "autoEvents": [{}],
      "protocols": {
        "ovms": {
          "Host": "192.168.198.8",
          "Model": "ssdlite_mobilenet_v2",
          "Port": 9000,
          "Record": false,
          "Snapshot": false,
          "Uri": "rtsp://192.168.123.12:18554/test",
          "Version": 1,
          "Score": 0.3
        }
      }
    }
  ]
}

验证配置

所有的容器都运行成功后，可看到类似以下的结果：

# docker ps --format 'table {{.Image}}\t{{.Names}}'

IMAGE                                                NAMES
①
openvino/model_server:latest-gpu                     model_server

②
yiqisoft/YiMEDIA                                     yimedia

③
edgexfoundry/device-openvino-ssdlite-mobilenet-v2:0.0.0-dev       edgex-device-openvino-resnet
edgexfoundry/app-service-configurable:3.1.0                       edgex-app-rules-engine
edgexfoundry/core-data:3.1.0                                      edgex-core-data
edgexfoundry/core-command:3.1.0                                   edgex-core-command
edgexfoundry/support-scheduler:3.1.0                              edgex-support-scheduler
edgexfoundry/core-common-config-bootstrapper:3.1.0                edgex-core-common-config-bootstrapper
edgexfoundry/support-notifications:3.1.0                          edgex-support-notifications
edgexfoundry/core-metadata:3.1.0                                  edgex-core-metadata
eclipse-mosquitto:2.0.18                                          edgex-mqtt-broker
edgexfoundry/edgex-ui:3.1.0                                       edgex-ui-go
redis:7.0.14-alpine                                               edgex-redis
hashicorp/consul:1.16.2                                           edgex-core-consul

OpenVINO Model Server 日志

# docker logs -f model_server

model_server  | [2024-04-11 04:29:29.941][131][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.022 ms
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.943][131][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.492 ms
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.547 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.226 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 10.4070 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.850 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.953][128][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.192 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.519 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.052 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 12.787 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.004 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.976][132][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.705 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.461 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.049 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 10.4050 ms

可以看到推理使用设备：Used device: CPU，也可以在 config.json 配置文件中修改为： AUTO 或 GPU
整个请求所花的时间：Total gRPC request processing time: 10.4050 ms，综合平均值，也就是10-15ms 之间，性能非常好。

推理结果

通过 Web MJPEG 方式实时查看推理结果，例如： http://localhost:18080/OpenVINO-HK2

上海亿琪软件有限公司，国家级高新技术企业，全球开放边缘计算和物联网领域的领导者，全球领先的工业物联网软件开发商和解决方案提供商，助力企业和组织实现数字化转型。公司专注于 5G 通信、AI 人工智能、边缘计算和大数据网络安全多项技术领域，致力于物联网领域前沿技术的创新，为用户提供全方位、智能化和安全的物联网解决方案。

2023 年，公司发布“ YiFUSION ｜工业边缘智能融合网关 ”产品，为工业客户提供一整套的边缘计算+AI 能力：高性能数据采集、多类型数据融合、AI 算法集成、云端业务对接。在边缘网关的基础上，集成了 IoT 平台的边缘协同能力、本地 Web SCADA 和 HMI 功能、本地数据存储、边缘 AI 视频分析、行业应用集成等。
2022 年，公司推出 “ YiCLOUD ｜亿琪云 ”一站式物联网应用解决方案。公司的业务涵盖了智慧城市、智慧农业、智能工厂和智慧园区等多个领域，公司软硬件产品和解决方案获得华为技术认证，得到中国移动 OCP 认证，公司还是边缘计算产业联盟 ECC 成员。

上海亿琪软件有限公司

联系方式：

网站：www.yiqisoft.cn

电话：18616669123

Email：jiekechoo@yiqisoft.cn