feature: add hunyuan llm support for plugins/ai-proxy #1018
Conversation
|
|
|
无法测通是什么意思？报什么错，gateway 容器输出什么 |
|
你是用的最新的代码吗？参考一下这个 PR 里的文档：#1005 |
嗯，代码是最新的，这个感谢今天上午的提醒，我先参考这个编译方案试一试 |
9458869
to
0a17538
Compare
|
Ⅰ. Describe what this PR did # File generated by hgctl. Modify as required.
admin:
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
scheme_header_transformation:
scheme_to_overwrite: https
stat_prefix: ingress_http
# Output envoy logs to stdout
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
# Modify as required
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: [ "*" ]
routes:
- match:
prefix: "/"
route:
cluster: moonshot
timeout: 300s
http_filters:
- name: wasmtest
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
name: wasmtest
vm_config:
runtime: envoy.wasm.runtime.v8
code:
local:
filename: /etc/envoy/plugin.wasm
configuration:
"@type": "type.googleapis.com/google.protobuf.StringValue"
value: |
{
"provider": {
"type": "hunyuan",
"hunyuanAuthKey": "VuR92ugGi04yr0EezLe7lm0FiKzrw27N",
"apiTokens": [
"sk-YGeSIaMRA2oSaDa86NCBVPGKdaiSuQ0YSOGI3nEkfvSb4HdT"
],
"hunyuanAuthId": "AKID2669UvMvTMJF86HbuMnB1rmdZTEvY2KQ",
"timeout": 1200000,
"modelMapping": {
"*": "hunyuan-lite"
}
}
}
- name: envoy.filters.http.router
clusters:
- name: httpbin
connect_timeout: 30s
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: httpbin
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: httpbin
port_value: 80
- name: moonshot
connect_timeout: 30s
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: moonshot
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: hunyuan.tencentcloudapi.com
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
"sni": "hunyuan.tencentcloudapi.com"使用如下docker-compose启动该插件： version: '3.7'
services:
envoy:
image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:1.4.0
entrypoint: /usr/local/bin/envoy
# 注意这里对wasm开启了debug级别日志，正式部署时则默认info级别
command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
depends_on:
- httpbin
networks:
- wasmtest
ports:
- "10000:10000"
volumes:
- ./envoy.yaml:/etc/envoy/envoy.yaml
- ./out/plugin.wasm:/etc/envoy/plugin.wasm
httpbin:
image: kennethreitz/httpbin:latest
networks:
- wasmtest
ports:
- "12345:80"
networks:
wasmtest: {}请求样例如下： curl --location 'http://127.0.0.1:10000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
"model": "gpt-3",
"messages": [
{
"role": "system",
"content": "你是一个名专业的开发人员！"
},
{
"role": "user",
"content": "你好，你是谁？"
}
],
"temperature": 0.3,
"stream": false
}'而后你就可以在本地的pod中查看相应的日志输出： |
|
hi，我已经按照意见进行了修改，请再review一下吧^v^ @CH3CHO |
|
hi，我已经按照意见进行了修改，请辛苦再review一下吧^v^ @CH3CHO |
麻烦按照上面的提示签署一下 CLA。 @xychen5 |
done，感谢review~ |
给出了本地测试千问的配置文件，发现无法测通，请求url是：curl --location 'http://127.0.0.1:10000/v1/chat/completions'
--header 'Content-Type: application/json'
--data '{
"model": "gpt-4-turbo",
"messages": [
{
"role": "user",
"content": "你是谁？"
}
],
"temperature": 0.3
}'