Kule blog, thina ukwakha ukubuyekeza ukubuyekeza ukubuyekeza ukubuyekeza ngokuvamile. Ngokwesibonelo, ungakwazi ukubuyekeza "i-elephant", noma "i-animal enhle" nge-list ye-images njenge-input. Thina usebenzisa model embedding multimodal ukucacisa futhi ukucacisa umbhalo, futhi ukwakha index ye-vector yokufaka ngokushesha. Thina usebenzisa CocoIndex ukwakha flow indexing, kuyinto ultra ukusebenza real-time data transformation framework. Ngesikhathi esebenza, ungakwazi ukongeza amafayela amasha ifolda futhi kusebenza kuphela amafayela ezintsha futhi iyatholakala ku-indexed ngaphakathi imizuzu. Ukubaluleka kakhulu kwethu uma ungahambisa umkhosi , uma lokhu tutorial iyahambisana. I-CocoIndex ku-GitHub Imikhiqizo I-CocoIndex is a ultra-performance real-time data transformation framework for AI. I-CocoIndex I-Clip ye-VIT-L/14 is a powerful vision-language model that can understand both images and texts. It is trained to align visual and textual representations in a shared embedding space, okwenza it ephelele for yethu image search usage case. I-Clip ye-VIT-L/14 Ngo-project yethu, sicela usebenzisa i-Clip ukuze: Ukuguqulwa kwe-embeddings ze-images ngqo Ukuguqulwa kwe-natural language search requests ku-embedding space efanayo Ukuvumela ukubuyekeza kwe-semantic ngokuhlanganisa ukubuyekeza kwe-query ne-subtitle Ngena ngemvume is a high-performance vector database. Thina usebenzisa ukubheka futhi ukubheka imibhalo. Ngena ngemvume Ukuhlobisa is a modern, fast (high-performance), web framework for building APIs with Python 3.7+ based on standard Python type hints. Thina usebenzisa ukuba ukwakha web API for image search. Ukuhlobisa Izinzuzo Ngena Postgres. I-CocoIndex isetshenziselwa i-Postgres ukucubungula i-lineage yedatha ukuze kusetshenziswe ngokushesha. Ukufaka Qdrant. Ukubonisa Indexing Flow Ukuhlobisa Flow I-flow diagram ibonisa indlela yokusebenza kwe-codebase yethu: Ukuhlola amafayela ze-image kusuka ku-local file system Ukusebenzisa i-Clip ukufumana nokufaka i-image Thola ama-embeddings ku-vector database yokufaka 1. Ingxubevange imifanekiso. @cocoindex.flow_def(name="ImageObjectEmbedding") def image_object_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope): data_scope["images"] = flow_builder.add_source( cocoindex.sources.LocalFile(path="img", included_patterns=["*.jpg", "*.jpeg", "*.png"], binary=True), refresh_interval=datetime.timedelta(minutes=1) # Poll for changes every 1 minute ) img_embeddings = data_scope.add_collector() Yenza ithebula nge subfields ( Ngena ngemva ), singatholakala ku- Ukuze uthole okwengeziwe flow_builder.add_source filename content Umhlahlandlela 2. Ukusebenza zonke imifanekiso futhi ukuthatha ulwazi. 2.1 Ukukhishwa kwe-image nge-Clip @functools.cache def get_clip_model() -> tuple[CLIPModel, CLIPProcessor]: model = CLIPModel.from_pretrained(CLIP_MODEL_NAME) processor = CLIPProcessor.from_pretrained(CLIP_MODEL_NAME) return model, processor Waze I-decorator ibhokisi imiphumela ye-function call. Kulesi sikhathi, ivimbele ukuthi sinikeza i-CLIP model kanye ne-processor nje. @functools.cache @cocoindex.op.function(cache=True, behavior_version=1, gpu=True) def embed_image(img_bytes: bytes) -> cocoindex.Vector[cocoindex.Float32, Literal[384]]: """ Convert image to embedding using CLIP model. """ model, processor = get_clip_model() image = Image.open(io.BytesIO(img_bytes)).convert("RGB") inputs = processor(images=image, return_tensors="pt") with torch.no_grad(): features = model.get_image_features(**inputs) return features[0].tolist() is a custom function that uses the CLIP model to convert an image into a vector embedding. It akutholele idatha image in bytes format and returns a list of floating-point numbers representing the image's embedding. embed_image Umsebenzi inikeza caching ngokusebenzisa I-parameter. Uma i-enabled, i-executor iyathayela imiphumela ye-function ukuze isetshenziswe ngemva kwe-reprocessing, okufanayo ikakhulukazi ngezinsizakalo ezinzima. Ukuze uthole okwengeziwe mayelana ne-parameter ye-function eyenziwe, bheka ku . cache Umhlahlandlela Ngemuva kwalokho, siza kusetshenziswe zonke imifanekiso kanye nokufaka ulwazi. with data_scope["images"].row() as img: img["embedding"] = img["content"].transform(embed_image) img_embeddings.collect( id=cocoindex.GeneratedField.UUID, filename=img["filename"], embedding=img["embedding"], ) 2.3 Ukukhuphela Izikhwama I-Export ye-embeddings ku-table ku-Qdrant. img_embeddings.export( "img_embeddings", cocoindex.storages.Qdrant( collection_name="image_search", grpc_url=QDRANT_GRPC_URL, ), primary_key_fields=["id"], setup_by_user=True, ) 3. Qhubeka i-index Incubate isibuyekezo nge-CLIP, okuyinto ibheka umbhalo kanye nezithombe kwelinye indawo yokubuyekeza, okuvumela ukubuyekeza kwezibuyekezo ze-cross-modal. def embed_query(text: str) -> list[float]: model, processor = get_clip_model() inputs = processor(text=[text], return_tensors="pt", padding=True) with torch.no_grad(): features = model.get_text_features(**inputs) return features[0].tolist() Ukubonisa FastAPI endpoint kusebenza ku-semantic image search. /search @app.get("/search") def search(q: str = Query(..., description="Search query"), limit: int = Query(5, description="Number of results")): # Get the embedding for the query query_embedding = embed_query(q) # Search in Qdrant search_results = app.state.qdrant_client.search( collection_name="image_search", query_vector=("embedding", query_embedding), limit=limit ) Ukusuka ku-Qdrant database ye-vector for similar embeddings. Ukuguqulwa kwe-top imiphumela limit # Format results out = [] for result in search_results: out.append({ "filename": result.payload["filename"], "score": result.score }) return {"results": out} Kuyinto endpoint kuvumela ukubuyekeza imifanekiso semantic lapho abasebenzisi angakwazi ukufumana imifanekiso ngokuvamile, ngaphandle kokusebenzisa i-keyword matchings esifanele. isicelo isikhunta eside app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Serve images from the 'img' directory at /img app.mount("/img", StaticFiles(directory="img"), name="img") Ukulungiswa kwe-application ye-FastAPI nge-middleware ye-CORS kanye ne-file ye-static enikezelwe ku: Ukuvumela izicelo ze-cross-origin kusuka kumazwe eyodwa Ukusebenza amafayela ze-image ye-static kusuka ku- 'img' directory Hlola i-API endpoints for image search functionality @app.on_event("startup") def startup_event(): load_dotenv() cocoindex.init() # Initialize Qdrant client app.state.qdrant_client = QdrantClient( url=QDRANT_GRPC_URL, prefer_grpc=True ) app.state.live_updater = cocoindex.FlowLiveUpdater(image_object_embedding_flow) app.state.live_updater.start() Umshini we-start event utshintshe isicelo lapho kuqala kuqala. Ngiyazi ukuthi zonke izigaba zenza: load_dotenv(): I-environment variables ifayilishwe kusuka ku-env ifayela, okuyinto ezisebenzayo ukubuyekeza njenge-API keys ne-URLs cocoindex.init(): I-CocoIndex framework ifumaneka, ukwakha izingxenye nezidingo kanye nezinhlangano Qdrant Client Setup: Creates a new instance QdrantClient Configures it to use the gRPC URL specified in environment variables Enables gRPC preference for better performance Stores the client in the FastAPI app state for access across requests Live Updater Setup: Creates a instance for the FlowLiveUpdater image_object_embedding_flow This enables real-time updates to the image search index Starts the live updater to begin monitoring for changes Ukulungiswa okuhlobene kuncike ukuthi zonke izingxenye ezidingekayo zihlanganiswa ngokufanelekileyo futhi zokusebenza lapho isicelo kuqala. Ukubuyekezwa ungakwazi ukuyifaka i-frontend code . Thina ngempumelelo ukugcina kulula futhi minimalistic ukucindezeleka ku-image search functionality. Ngiya isikhathi eside ukujabulela! Create a collection in Qdrant curl -X PUT 'http://localhost:6333/collections/image_search' \ -H 'Content-Type: application/json' \ -d '{ "vectors": { "embedding": { "size": 768, "distance": "Cosine" } } }' Setup indexing flow cocoindex setup main.py It is setup with a live updater, so you can add new files to the folder and it will be indexed within a minute. Run backend uvicorn main:app --reload --host 0.0.0.0 --port 8000 Run frontend cd frontend npm install npm run dev Thola Ukubuyiselwa http://localhost:5174 Ngiyaxolisa isithombe esilandelayo ku- Folders, isibonelo, lokhu , noma umbhalo owaziwa. Qinisekisa imizuzu yokusebenza kanye nokuguqulwa kwebhizinisi entsha. img Izingubo ezinhle Uma ufuna ukucubungula ukucubungula, ungakwazi ukucubungula ku-CocoInsight . cocoindex server -ci main.py Finally - we are constantly improving, and more features and examples are coming soon. If you love this article, please give us a star ⭐ at to help us grow. Thanks for reading! GitHub Ngena ngemvume
