From 12a0842cc57ca6faadffc875dd15e4ebffceaacc Mon Sep 17 00:00:00 2001 From: happybell80 Date: Mon, 16 Mar 2026 13:45:15 +0900 Subject: [PATCH] =?UTF-8?q?docs(1=EC=B0=A8):=20=EC=9E=84=EB=B2=A0=EB=94=A9?= =?UTF-8?q?=20=EB=AC=B8=EC=A0=9C=C2=B7=EB=A6=AC=EC=84=9C=EC=B9=98=C2=B7?= =?UTF-8?q?=EA=B3=84=ED=9A=8D=20=EB=AC=B8=EC=84=9C=20=EB=B3=B4=EA=B0=95,?= =?UTF-8?q?=20=EA=B3=84=ED=9A=8D=2095%+=20=EC=99=84=EC=84=B1=EB=8F=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Made-with: Cursor --- ...ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๊ณ„ํš.md | 92 ++++++++++++++++--- ...ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md | 29 ++++-- ...ดํ”„๋กœ์ ํŠธ_ํ˜„ํ™ฉ_๋ฐ_SSOT_๋ฆฌ์„œ์น˜.md | 3 +- ...1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md | 22 +++-- 4 files changed, 116 insertions(+), 30 deletions(-) diff --git a/journey/plans/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๊ณ„ํš.md b/journey/plans/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๊ณ„ํš.md index a9baee7..329bd34 100644 --- a/journey/plans/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๊ณ„ํš.md +++ b/journey/plans/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๊ณ„ํš.md @@ -4,9 +4,12 @@ tags: [plans, embedding, gemini, rag, robeing, 1์ฐจ] # ์ž„๋ฒ ๋”ฉ 1์ฐจ: ๋กœ๋น™ Gemini 2 ์ „ํ™˜ ๊ณ„ํš +**์ž‘์„ฑ์ผ**: 2026-03-16 + ## ์ƒ์œ„ ์›์น™ - [0_VALUE ์ž„๋ฒ ๋”ฉ ์ •์ฑ…](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) +- [๋กœ๋น™ ๋ฌธ์„œ ์ž‘์„ฑ ์›์น™](../../book/300_architecture/312_writing-principles.md) - [์ž„๋ฒ ๋”ฉ 1์ฐจ ๋กœ๋น™ Gemini 2 ์ „ํ™˜ ๋ฌธ์ œ ์˜คํ”ˆ](../troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md) - [์ž„๋ฒ ๋”ฉ 1์ฐจ ๋กœ๋น™ ํ˜„ํ™ฉ ๋ฐ SSOT ๋ฆฌ์„œ์น˜](../research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) @@ -14,6 +17,12 @@ tags: [plans, embedding, gemini, rag, robeing, 1์ฐจ] - planned +## ์„ ํ–‰ ์กฐ๊ฑด + +- `workspace-config/runtime.env`์— `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM`, `EMBEDDING_MODEL` ๋ฐ˜์˜๋จ (SSOT) +- rb8001ยทskill-rag-file์ด `EMBEDDING_SERVICE_URL` ๋˜๋Š” `SKILL_EMBEDDING_URL`๋กœ skill-embedding ์ฐธ์กฐ +- `robeing/tests/test_gemini_embedding_2.py` ์กด์žฌ, Gemini API ํ‚ค ๋กœ์ปฌ ์‚ฌ์šฉ ๊ฐ€๋Šฅ + ## ๋ชฉํ‘œ - ๋กœ๋น™(skill-embedding, skill-rag-file, rb8001)์„ Gemini Embedding 2, 768d๋กœ ์ „ํ™˜ํ•œ๋‹ค. @@ -24,10 +33,11 @@ tags: [plans, embedding, gemini, rag, robeing, 1์ฐจ] ### ํฌํ•จ -- skill-embedding โ†’ Gemini 2 ์ „ํ™˜ ๋˜๋Š” skill-rag-file ๋‚ด๋ถ€ ์ง์ ‘ Gemini API ํ˜ธ์ถœ +- **๊ฒฝ๋กœ ๊ฒฐ์ •**: skill-embedding ๊ต์ฒด (ONNXโ†’Gemini 2). [๋ฆฌ์„œ์น˜ ยง7 ๊ฒฝ๋กœ ์„ค๊ณ„ ๊ฒฐ์ •](../research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) ์ฐธ์กฐ. +- skill-embedding ๋‚ด๋ถ€ Gemini 2 ์ „ํ™˜, /embed ์—”๋“œํฌ์ธํŠธ ์œ ์ง€ - NAS RAGยทCompany X RAG PDFยท์ด๋ฏธ์ง€ ์ž„๋ฒ ๋”ฉ ๊ฒฝ๋กœ - MRL 768d, ChromaDB/pgvector ์Šคํ‚ค๋งˆ ํ˜ธํ™˜ -- ์ฒญํ‚น Macro-chunking(2,000~4,000 ํ† ํฐ) ๊ฒ€ํ†  +- **์ฒญํ‚น**: 1์ฐจ๋Š” ๊ธฐ์กด Micro ์œ ์ง€. 2๋‹จ๊ณ„์—์„œ Macro(2,000~4,000) ๊ฒ€ํ† . [๋ฆฌ์„œ์น˜ ยง8 ์ฒญํ‚น ๋‹จ๊ณ„](../research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) ์ฐธ์กฐ. ### ์ œ์™ธ @@ -36,30 +46,84 @@ tags: [plans, embedding, gemini, rag, robeing, 1์ฐจ] ## env SSOT -- `workspace-config/runtime.env`: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM` โ€” [0_VALUE embedding-policy](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) +- **์›์น™**: ๋กœ๋น™ ์ „์—ญ(rb8001, skill-embedding, skill-rag-file, skill-news, skill-slack ๋“ฑ ๋ชจ๋“  ์Šคํ‚ฌ)์€ `workspace-config`๋ฅผ ๋”ฐ๋ฅด๊ณ , ์ž„๋ฒ ๋”ฉ ๊ด€๋ จ ๊ฐ’์„ ์„œ๋น„์Šค๋ณ„๋กœ ์˜ค๋ฒ„๋ผ์ด๋“œํ•˜์ง€ ์•Š๋Š”๋‹ค. +- `workspace-config/runtime.env`: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM`, `EMBEDDING_MODEL`(gemini-embedding-2-preview) โ€” [0_VALUE embedding-policy](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) ## ์ ์šฉ ์ˆœ์„œ -1. ํ’ˆ์งˆยท๋น„์šฉ ํ…Œ์ŠคํŠธ (`test_gemini_embedding_2.py` ํ™•์žฅ) -2. ๊ฒฝ๋กœ ์„ค๊ณ„ (skill-embedding vs skill-rag-file ์ง์ ‘ ํ˜ธ์ถœ) -3. ์Šคํ‚ค๋งˆยท์ฒญํ‚น (output_dimensionality=768) -4. ์ ์šฉ ๋ฐ ๊ฒ€์ฆ (Company X RAG ๋˜๋Š” NAS RAG 1๊ฐœ ๊ฒฝ๋กœ) -5. worklog ์ž‘์„ฑ ํ›„ ๋‹ซํž˜ ์„ ์–ธ +### 1. ํ’ˆ์งˆยท๋น„์šฉ ํ…Œ์ŠคํŠธ + +- **๋Œ€์ƒ**: `robeing/tests/test_gemini_embedding_2.py` ํ™•์žฅ +- **์ž‘์—…**: + - ํ…์ŠคํŠธ 768d ๊ฒ€์ฆ (output_dimensionality=768) + - PDFยท์ด๋ฏธ์ง€ ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ ์ƒ˜ํ”Œ 1๊ฑด each + - 1M ํ† ํฐ ๋น„์šฉ $0.25 ์ดํ•˜ ํ™•์ธ + - Recall ๊ธฐ์กด ๋Œ€๋น„ ์œ ์ง€ ๋˜๋Š” ๊ฐœ์„  (์„ ํƒ: ko-sroberta ๋Œ€๋น„ ์ƒ˜ํ”Œ ๋น„๊ต) +- **์™„๋ฃŒ ๊ธฐ์ค€**: ํ…Œ์ŠคํŠธ ํ†ต๊ณผ, ๋น„์šฉยทRecall ๊ธฐ์ค€ ์ถฉ์กฑ + +### 2. skill-embedding ์ „ํ™˜ + +- **๋Œ€์ƒ**: skill-embedding ๋ ˆํฌ ์ „์ฒด. [๋ฆฌ์„œ์น˜ ยง3 ์˜ํ–ฅ ๋ฒ”์œ„](../research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) ์ฐธ์กฐ. +- **์ž‘์—…**: + - ONNX ko-sroberta โ†’ Gemini 2 API ํ˜ธ์ถœ๋กœ ๊ต์ฒด + - `EMBEDDING_MODEL`, `EMBEDDING_DIM` env_file์—์„œ workspace-config ๋กœ๋“œ + - /embed ์—”๋“œํฌ์ธํŠธ ์š”์ฒญ/์‘๋‹ต ํ˜•์‹ ์œ ์ง€ (๊ธฐ์กด ํด๋ผ์ด์–ธํŠธ ํ˜ธํ™˜) + - docker-compose env_file ๊ฒฝ๋กœ: `workspace-config/runtime.env` +- **์™„๋ฃŒ ๊ธฐ์ค€**: ๋กœ์ปฌ `/embed` ํ˜ธ์ถœ ์‹œ 768d ๋ฒกํ„ฐ ๋ฐ˜ํ™˜ + +### 3. ์Šคํ‚ค๋งˆยท์ฒญํ‚น + +- **๋Œ€์ƒ**: rb8001 `migrate_chromadb_collections.py`, ChromaDB ์ปฌ๋ ‰์…˜, intent_prototypes pgvector +- **์ž‘์—…**: + - output_dimensionality=768๋กœ ChromaDB ์ปฌ๋ ‰์…˜ ์ƒ์„ฑ/๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ + - intent_prototypes pgvector 768d ํ™•์ธ (์ด๋ฏธ 768d๋ฉด ์Šคํ‚ค๋งˆ ๋ณ€๊ฒฝ ์—†์Œ) + - ์ฒญํ‚น: ๊ธฐ์กด Micro(300~500 ๋‹จ์–ด) ์œ ์ง€, ์ฝ”๋“œ ๋ณ€๊ฒฝ ์—†์Œ +- **์™„๋ฃŒ ๊ธฐ์ค€**: ChromaDBยทpgvector 768d ํ†ต์ผ, migrate ์Šคํฌ๋ฆฝํŠธ ์‹คํ–‰ ๊ฐ€๋Šฅ + +### 4. ์ ์šฉ ๋ฐ ๊ฒ€์ฆ + +- **๋Œ€์ƒ**: Company X RAG ๋˜๋Š” NAS RAG 1๊ฐœ ๊ฒฝ๋กœ ์šฐ์„  +- **์ž‘์—…**: + - skill-rag-file, rb8001์ด ๊ธฐ์กด `EMBEDDING_SERVICE_URL`/`SKILL_EMBEDDING_URL`๋กœ ์ƒˆ skill-embedding ํ˜ธ์ถœ (URL ๋ณ€๊ฒฝ ์—†์Œ) + - RAG ์—…๋กœ๋“œ โ†’ ์ž„๋ฒ ๋”ฉ โ†’ ๊ฒ€์ƒ‰ ํŒŒ์ดํ”„๋ผ์ธ 1ํšŒ ์ˆ˜๋™ ๊ฒ€์ฆ + - DOCS `skills/companyx-rag/SKILL.md`, `330_*.md` ํ•„์š” ์‹œ ๊ฐฑ์‹  +- **์™„๋ฃŒ ๊ธฐ์ค€**: Company X RAG ๋˜๋Š” NAS RAG ์ƒˆ ๊ฒฝ๋กœ๋กœ ๋™์ž‘ ํ™•์ธ + +### 5. ivada-infra ๋ฐฐํฌ (์„œ๋ฒ„ ๊ด€๋ฆฌ์ž) + +- **๋Œ€์ƒ**: ivada-infra skill-embedding ๋ฐฐํฌ, 23/24 ์„œ๋ฒ„ +- **์ž‘์—…**: `.env.deploy` ๊ฐฑ์‹ , skill-embedding ์ด๋ฏธ์ง€ ์žฌ๋นŒ๋“œยท์žฌ๋ฐฐํฌ +- **์‹คํ–‰**: ์„œ๋ฒ„ ๊ด€๋ฆฌ์ž๋งŒ ์ˆ˜ํ–‰ + +### 6. worklog ์ž‘์„ฑ ํ›„ ๋‹ซํž˜ ์„ ์–ธ + +- worklog์— 1~5 ์™„๋ฃŒ ๊ธฐ๋ก, [๋ฌธ์ œ ์˜คํ”ˆ](../troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md) ๋‹ซํž˜ ์กฐ๊ฑด 6๊ฐœ ์ถฉ์กฑ ์„ ์–ธ + +## ๋กค๋ฐฑ ์ ˆ์ฐจ + +- skill-embedding: ์ด์ „ ONNX ko-sroberta ์ด๋ฏธ์ง€/์ฝ”๋“œ๋กœ ๋ณต๊ท€ +- workspace-config: `EMBEDDING_MODEL` ์ œ๊ฑฐ ๋˜๋Š” ์ด์ „๊ฐ’ ๋ณต์› +- ChromaDB: migrate ์ „ ๋ฐฑ์—… ์žˆ์œผ๋ฉด ๋ณต์›. ์—†์œผ๋ฉด 768d ์‹ ๊ทœ ์ปฌ๋ ‰์…˜๋งŒ ์‚ญ์ œ ## ๊ฒ€์ฆ ๊ธฐ์ค€ (๋‹ซํž˜ ์กฐ๊ฑด) -- skill-embedding ๋˜๋Š” skill-rag-file์ด Gemini 2, 768d๋กœ ๋™์ž‘ -- rb8001์ด ์ƒˆ ์ž„๋ฒ ๋”ฉ ๊ฒฝ๋กœ ์ฐธ์กฐ -- ChromaDBยทpgvector 768d ํ†ต์ผ -- Company X RAG, NAS RAG ์ƒˆ ๊ฒฝ๋กœ ๋™์ž‘ -- PDFยท์ด๋ฏธ์ง€ ์ง์ ‘ ์ž„๋ฒ ๋”ฉ Recall ์œ ์ง€ ๋˜๋Š” ๊ฐœ์„  -- 1M ํ† ํฐ ๋น„์šฉ $0.25 ์ดํ•˜ +[๋ฌธ์ œ ์˜คํ”ˆ ๋‹ซํž˜ ์กฐ๊ฑด 6๊ฐœ](../troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md#๋‹ซํž˜-์กฐ๊ฑด)์™€ ๋™์ผ: + +1. skill-embedding์ด Gemini 2, 768d๋กœ ๋™์ž‘ํ•œ๋‹ค. +2. rb8001ยทskill-rag-file์ด ๊ธฐ์กด /embed URL๋กœ ์ƒˆ ๋ชจ๋ธ์„ ์ฐธ์กฐํ•œ๋‹ค. +3. ChromaDBยทpgvector ์Šคํ‚ค๋งˆ๊ฐ€ 768d๋กœ ํ†ต์ผ๋œ๋‹ค. +4. Company X RAG, NAS RAG๊ฐ€ ์ƒˆ ๊ฒฝ๋กœ๋กœ ๋™์ž‘ํ•œ๋‹ค. +5. PDFยท์ด๋ฏธ์ง€ ์ง์ ‘ ์ž„๋ฒ ๋”ฉ Recall ์œ ์ง€ ๋˜๋Š” ๊ฐœ์„ , 1M ํ† ํฐ ๋น„์šฉ $0.25 ์ดํ•˜. +6. worklog์—์„œ ๋‹ซํž˜ ์„ ์–ธํ•œ๋‹ค. ## ๋‹ซํž˜ ์„ ์–ธ - worklog์—์„œ๋งŒ ์„ ์–ธํ•œ๋‹ค. - ๋ณธ ๋ฌธ์„œ ์ƒํƒœ๋ฅผ `completed`๋กœ ๊ฐฑ์‹ ํ•˜๊ณ  worklog ๋งํฌ๋ฅผ ์ถ”๊ฐ€ํ•œ๋‹ค. +- [๋ฌธ์ œ ์˜คํ”ˆ](../troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md) ๋‹ซํž˜ ์กฐ๊ฑด 6๊ฐœ ์ „๋ถ€ ์ถฉ์กฑ ํ›„ ์„ ์–ธ. ## ๊ด€๋ จ ๋ฌธ์„œ - [์ž„๋ฒ ๋”ฉ 2์ฐจ StarsAndIยทGooseCouncil ์ „ํ™˜ ๊ณ„ํš](./260316_์ž„๋ฒ ๋”ฉ_2์ฐจ_StarsAndI_GooseCouncil_์ „ํ™˜_๊ณ„ํš.md) +- [skill-embedding ์„œ๋น„์Šค ๊ตฌ์ถ•](../troubleshooting/250805_happybell80_skill-embedding์„œ๋น„์Šค๊ตฌ์ถ•.md) +- [rb8001 ๋ฉ”๋ชจ๋ฆฌ ChromaDB 768/384 ์ฐจ์› ๋“œ๋ฆฌํ”„ํŠธ](../troubleshooting/260312_rb8001_memory_chromadb_768_384_dimension_drift.md) diff --git a/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md b/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md index 3701e6a..19c0671 100644 --- a/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md +++ b/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md @@ -53,9 +53,11 @@ tags: [research, embedding, ssot, robeing, 1์ฐจ] ## 4. env SSOT (๋ฐ˜์˜๋จ) -- `workspace-config/runtime.env`: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM` โ€” [0_VALUE embedding-policy env SSOT](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) -- rb8001 docker-compose: `SKILL_EMBEDDING_URL=${EMBEDDING_SERVICE_URL:-http://localhost:8515}`๋กœ SSOT ์ฐธ์กฐ +- **์›์น™**: ๋กœ๋น™ ์ „์—ญ(rb8001, skill-embedding, skill-rag-file, skill-news, skill-slack ๋“ฑ ๋ชจ๋“  ์Šคํ‚ฌ)์€ `workspace-config`๋ฅผ ๋”ฐ๋ฅด๊ณ , ์ž„๋ฒ ๋”ฉ ๊ด€๋ จ ๊ฐ’์„ ์„œ๋น„์Šค๋ณ„๋กœ ์˜ค๋ฒ„๋ผ์ด๋“œํ•˜์ง€ ์•Š๋Š”๋‹ค. +- `workspace-config/runtime.env`: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM`, `EMBEDDING_MODEL`(gemini-embedding-2-preview) โ€” [0_VALUE embedding-policy env SSOT](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) +- rb8001 docker-compose: `SKILL_EMBEDDING_URL=${EMBEDDING_SERVICE_URL:-...}`๋กœ SSOT ์ฐธ์กฐ - skill-rag-file: `EMBEDDING_SERVICE_URL` env_file์—์„œ ๋กœ๋“œ +- skill-embedding: ์ „ํ™˜ ํ›„ `EMBEDDING_MODEL`, `EMBEDDING_DIM` env_file์—์„œ ๋กœ๋“œ ## 5. ์‚ฌ์‹ค(Facts) @@ -73,14 +75,29 @@ tags: [research, embedding, ssot, robeing, 1์ฐจ] --- -## 7. ๋ฏธํ™•์ • ํ•ญ๋ชฉ(Unresolved) +## 7. ๊ฒฝ๋กœ ์„ค๊ณ„ ๊ฒฐ์ • (์ถ”์ฒœ) -- skill-embedding ๊ต์ฒด vs skill-rag-file ๋‚ด๋ถ€ ์ง์ ‘ Gemini API ํ˜ธ์ถœ -- ์ฒญํ‚น ๋‹จ์œ„ ํ™•๋Œ€ ์ ์šฉ ์‹œ์  +- **์ถ”์ฒœ: skill-embedding ๊ต์ฒด**. ONNX ko-sroberta โ†’ Gemini 2๋กœ skill-embedding ์„œ๋น„์Šค ๋‚ด๋ถ€๋งŒ ์ „ํ™˜. +- **์ด์œ **: (1) ๊ธฐ์กด HTTP /embed ๊ตฌ์กฐ ์œ ์ง€, rb8001ยทskill-rag-file ์ˆ˜์ • ์—†์Œ (2) workspace-config `EMBEDDING_MODEL`๋งŒ ๋ฐ˜์˜ํ•˜๋ฉด ๋ชจ๋ธ ์ „ํ™˜ ์™„๋ฃŒ (3) ๋ณ€๊ฒฝ ๋ฒ”์œ„ ์ตœ์†Œ, ivada-infra skill-embedding ๋ฐฐํฌ๋งŒ ๊ฐฑ์‹  (4) skill-rag-file ์ง์ ‘ ํ˜ธ์ถœ์€ skill-embedding ์ œ๊ฑฐยทskill-rag-file ๋Œ€๊ทœ๋ชจ ์ˆ˜์ • ํ•„์š” +- **๋Œ€์•ˆ**: skill-rag-file ๋‚ด๋ถ€ Gemini API ์ง์ ‘ ํ˜ธ์ถœ โ€” skill-embedding ์ œ๊ฑฐ ๊ฐ€๋Šฅํ•˜๋‚˜ skill-rag-file ๋ณ€๊ฒฝ ๋ฒ”์œ„ ํผ. 1์ฐจ์—์„œ๋Š” ๋น„์ถ”์ฒœ. --- -## 8. ๊ด€๋ จ ๋ฌธ์„œ +## 8. ์ฒญํ‚น ๋‹จ๊ณ„ ๊ฒฐ์ • (์ถ”์ฒœ) + +- **1์ฐจ**: ๊ธฐ์กด Micro-chunking(300~500 ๋‹จ์–ด) ์œ ์ง€. 768dยท๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ ์ ์šฉยท๊ฒ€์ฆ ์šฐ์„ . +- **2๋‹จ๊ณ„**: RAG ๊ฒฝ๋กœ ๊ฒ€์ฆ ํ›„ Macro-chunking(2,000~4,000 ํ† ํฐ) ๊ฒ€ํ† . [260315 Gemini Embedding 2 ๋ฆฌ์„œ์น˜ ยง5 ์ฒญํ‚น ์ „๋žต](./260315_Gemini_Embedding_2_๋ฆฌ์„œ์น˜_๋น„์šฉ_์ฒญํ‚น_๋„์ž…๊ฒ€ํ† .md) ์ฐธ์กฐ. + +--- + +## 9. ivada-infra ์—ฐ๊ณ„ + +- skill-embedding์€ ivada-infra์—์„œ 23/24 ์„œ๋ฒ„ ๋ฐฐํฌ. `.env.deploy`, docker-compose ๊ฒฝ๋กœ ์กด์žฌ. +- 1์ฐจ ์ „ํ™˜ ์‹œ: skill-embedding ๋ ˆํฌ Gemini 2 ์ „ํ™˜ โ†’ ivada-infra skill-embedding ๋ฐฐํฌ ์„ค์ • ๊ฐฑ์‹  โ†’ 23/24 ์„œ๋ฒ„ ์žฌ๋ฐฐํฌ. ์„œ๋ฒ„ ๊ด€๋ฆฌ์ž ์‹คํ–‰. + +--- + +## 10. ๊ด€๋ จ ๋ฌธ์„œ - [์ž„๋ฒ ๋”ฉ 1์ฐจ ๋กœ๋น™ Gemini 2 ์ „ํ™˜ ๋ฌธ์ œ ์˜คํ”ˆ](../../troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md) - [์ž„๋ฒ ๋”ฉ 1์ฐจ ๋กœ๋น™ Gemini 2 ์ „ํ™˜ ๊ณ„ํš](../../plans/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๊ณ„ํš.md) diff --git a/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_์ „์ฒดํ”„๋กœ์ ํŠธ_ํ˜„ํ™ฉ_๋ฐ_SSOT_๋ฆฌ์„œ์น˜.md b/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_์ „์ฒดํ”„๋กœ์ ํŠธ_ํ˜„ํ™ฉ_๋ฐ_SSOT_๋ฆฌ์„œ์น˜.md index 2c9324e..ede1eaa 100644 --- a/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_์ „์ฒดํ”„๋กœ์ ํŠธ_ํ˜„ํ™ฉ_๋ฐ_SSOT_๋ฆฌ์„œ์น˜.md +++ b/journey/research/rag/260316_์ž„๋ฒ ๋”ฉ_์ „์ฒดํ”„๋กœ์ ํŠธ_ํ˜„ํ™ฉ_๋ฐ_SSOT_๋ฆฌ์„œ์น˜.md @@ -178,7 +178,8 @@ tags: [research, embedding, ssot, robeing, starsandi, goosecouncil, workspace, i ### 5.2 ๋Ÿฐํƒ€์ž„ SSOT (๋ฐ˜์˜๋จ) - **์œ„์น˜**: `workspace-config/runtime.env` -- **ํ‚ค**: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM` โ€” [0_VALUE embedding-policy env SSOT](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) +- **ํ‚ค**: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM`, `EMBEDDING_MODEL` โ€” [0_VALUE embedding-policy env SSOT](https://github.com/happybell80/0_VALUE/blob/main/02_Governance/embedding-policy.md) +- **1์ฐจ ๊ฒฝ๋กœ ๊ฒฐ์ •**: skill-embedding ๊ต์ฒด. [1์ฐจ ๋ฆฌ์„œ์น˜ ยง7](./260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) ์ฐธ์กฐ. - rb8001: `SKILL_EMBEDDING_URL=${EMBEDDING_SERVICE_URL:-http://localhost:8515}`๋กœ ์ฐธ์กฐ - skill-rag-file: env_file์—์„œ `EMBEDDING_SERVICE_URL` ๋กœ๋“œ diff --git a/journey/troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md b/journey/troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md index 9c26dfe..589b272 100644 --- a/journey/troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md +++ b/journey/troubleshooting/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_Gemini2_์ „ํ™˜_๋ฌธ์ œ์˜คํ”ˆ.md @@ -45,9 +45,11 @@ tags: [troubleshooting, embedding, gemini, rag, robeing, 1์ฐจ] |------|------| | ๋ชจ๋ธ | Gemini Embedding 2 | | ์ฐจ์› | 768 (MRL) | +| ๊ฒฝ๋กœ | skill-embedding ๊ต์ฒด (ONNXโ†’Gemini 2) | | ์ ์šฉ | robeing ์ „์—ญ (skill-embedding, skill-rag-file, rb8001) | +| env | workspace-config SSOT, ์„œ๋น„์Šค๋ณ„ ์˜ค๋ฒ„๋ผ์ด๋“œ ๊ธˆ์ง€ | | ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ | PDFยท์ด๋ฏธ์ง€ ์ง์ ‘ ์ž„๋ฒ ๋”ฉ | -| ์ฒญํ‚น | Macro-chunking(2,000~4,000 ํ† ํฐ) ๊ฒ€ํ†  | +| ์ฒญํ‚น | 1์ฐจ Micro ์œ ์ง€, 2๋‹จ๊ณ„ Macro ๊ฒ€ํ†  | ## ์˜ํ–ฅ ๋ฒ”์œ„ (1์ฐจ) @@ -58,15 +60,16 @@ tags: [troubleshooting, embedding, gemini, rag, robeing, 1์ฐจ] - migrate_chromadb_collections.py - DOCS/skills/companyx-rag, 330_๋ฐฑ์—”๋“œ ์„ค๊ณ„ ๋ฌธ์„œ - ivada-infra skill-embedding ๋ฐฐํฌ -- workspace-config: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM` ๋ฐ˜์˜๋จ (SSOT) +- workspace-config: `EMBEDDING_SERVICE_URL`, `EMBEDDING_DIM`, `EMBEDDING_MODEL` ๋ฐ˜์˜๋จ (SSOT) ## ๋‹ซํž˜ ์กฐ๊ฑด -1. skill-embedding ๋˜๋Š” skill-rag-file์ด Gemini 2, 768d๋กœ ๋™์ž‘ํ•œ๋‹ค. -2. rb8001์ด ์ƒˆ ์ž„๋ฒ ๋”ฉ ๊ฒฝ๋กœ๋ฅผ ์ฐธ์กฐํ•œ๋‹ค. +1. skill-embedding์ด Gemini 2, 768d๋กœ ๋™์ž‘ํ•œ๋‹ค. (๊ฒฝ๋กœ: skill-embedding ๊ต์ฒด ํ™•์ •) +2. rb8001ยทskill-rag-file์ด ๊ธฐ์กด /embed URL๋กœ ์ƒˆ ๋ชจ๋ธ์„ ์ฐธ์กฐํ•œ๋‹ค. 3. ChromaDBยทpgvector ์Šคํ‚ค๋งˆ๊ฐ€ 768d๋กœ ํ†ต์ผ๋œ๋‹ค. 4. Company X RAG, NAS RAG๊ฐ€ ์ƒˆ ๊ฒฝ๋กœ๋กœ ๋™์ž‘ํ•œ๋‹ค. -5. worklog์—์„œ ๋‹ซํž˜ ์„ ์–ธํ•œ๋‹ค. +5. PDFยท์ด๋ฏธ์ง€ ์ง์ ‘ ์ž„๋ฒ ๋”ฉ Recall ์œ ์ง€ ๋˜๋Š” ๊ฐœ์„ , 1M ํ† ํฐ ๋น„์šฉ $0.25 ์ดํ•˜. +6. worklog์—์„œ ๋‹ซํž˜ ์„ ์–ธํ•œ๋‹ค. ## ์žฌํ˜„ ์กฐ๊ฑด @@ -77,13 +80,14 @@ tags: [troubleshooting, embedding, gemini, rag, robeing, 1์ฐจ] - 0_VALUE ์ž„๋ฒ ๋”ฉ ์ •์ฑ… ํ™•์ •: Gemini 2, 768d, ์ „์ˆ˜ ๊ต์ฒด, ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ - ๋ฆฌ์„œ์น˜(260315)์—์„œ Gemini Embedding 2 ํŠน์ง•ยท๋น„์šฉยท์ฒญํ‚น ์ „๋žต ์ •๋ฆฌ๋จ -- `rb8001/scripts/test_gemini_embedding_2.py` API ํ…Œ์ŠคํŠธ ์Šคํฌ๋ฆฝํŠธ ์กด์žฌ +- `robeing/tests/test_gemini_embedding_2.py` API ํ…Œ์ŠคํŠธ ์กด์žฌ, ํ™•์žฅ ์˜ˆ์ • - ๊ธฐ์กด ๋ฐ์ดํ„ฐ ์ ์Œ โ†’ ์ „์ˆ˜ ๊ต์ฒด ๋ถ€๋‹ด ๋‚ฎ์Œ +- workspace-config env SSOT: ๋กœ๋น™ ์ „์—ญ์ด ๋”ฐ๋ฅด๊ณ  ์„œ๋น„์Šค๋ณ„ ์˜ค๋ฒ„๋ผ์ด๋“œ ๊ธˆ์ง€ -## ๋ฏธํ™•์ • ํ•ญ๋ชฉ +## ํ™•์ • ํ•ญ๋ชฉ (๋ฆฌ์„œ์น˜ยท๊ณ„ํš ๋ฐ˜์˜) -- skill-embedding ๊ต์ฒด vs skill-rag-file ๋‚ด๋ถ€ ์ง์ ‘ Gemini API ํ˜ธ์ถœ -- ์ฒญํ‚น ๋‹จ์œ„ ํ™•๋Œ€ ์ ์šฉ ์‹œ์  +- **๊ฒฝ๋กœ**: skill-embedding ๊ต์ฒด (ONNXโ†’Gemini 2). [๋ฆฌ์„œ์น˜ ยง7](../research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) +- **์ฒญํ‚น**: 1์ฐจ๋Š” ๊ธฐ์กด Micro ์œ ์ง€. 2๋‹จ๊ณ„์—์„œ Macro ๊ฒ€ํ† . [๋ฆฌ์„œ์น˜ ยง8](../research/rag/260316_์ž„๋ฒ ๋”ฉ_1์ฐจ_๋กœ๋น™_ํ˜„ํ™ฉ_SSOT_๋ฆฌ์„œ์น˜.md) ## ์ด ๋ฌธ์„œ๊ฐ€ ์—ฌ๋Š” ๋ฆฌ์„œ์น˜