embedder_collection

(sometimes the results are more truthful if the “chat with document only” option is used) Incidentally, the Embedder model is only one part of a good RAG (Retrieval-Augmented Generation), but it should be tailored to your language and, if you want it to be completely accurate, also to the application, e.g. programming or medicine. Working well, all other its up to you! Some models are very similar! (jina and qwen based you can add manual to LM-Studio, set model "gear wheel" below "overide domain type") With the same setting, these embedders found same 6-7 snippets out of 10 from a book. This means that only 3-4 snippets were different, but I didn't test it extensively. Further tests have shown that the following models are suitable for complex tasks (German-text, but should be similar in English). Jina-DE, nomic was not that good. I'm not convinced by large models such as Qwen or JinaaiV3 and V4 doesnt work with LM studio; they are ten times slower and the result is not ten times better. Despite all this, you can recognize tables and some images. There are two embedder to find toxic content (toxic-prompt-roberta and minilmv2-toxic-jigsaw), dont know how good it works, and from ibm it give a whole LLM model (granite-guardian). ...

Short hints for using (Example for a large context with many expected hits):

Set your (Max Tokens)context-lenght 8000t main-LLM-model "LM-Studio with ALLM you must set also in LM-Stutio settings!", set your embedder-model (Max Embedding Chunk Length in character) 1500,set (Max Context Snippets, receiving) 10, in ALLM set also (Text splitting & Chunking Preferences - Text Chunk Size) 1500 character parts and (Search Preference) "accuracy". And set in your workspace 10 snippets. Hint in ALLM, set all in LM studio start both models and both are on top in ALLM.

-> Ok what that mean! Your document will be embedd in x times 1500 character/chunks(snippets), You can receive 10-snippets a 1500chars=15000chars (~4000token) from your document all in all -> ~2500words(2.5pages). now ~4000t left (from 8000t) for the answer ~2500words (2.5 pages). one question - one answer, with every setup i suggest only query once. If you change your settings so that you receive more than 8000t in the response, you should use a good LLM model to obtain a detailed and meaningful response from such a large amount of content. It doesn't matter whether the model can process 100'000 tokens; that doesn't mean that this model can process 100'000 meaningfully. You can play and set for your needs, eg 5-snippets a 5000chars, or 20-snippets a 500chars ... (every time you change the chunk-length the document must be embedd again). With these settings everything fits best for ONE answer. english vs german differ 50% in calculate tokens/word but ~5000 characters is one page of a book (no matter ger/en). But if you calculate with words ... words in german are longer, that means per word more token. The example is english, for german you can add apox 50% more token/word (1000 words ~1800t) Vector Size (Dimensions- you can not change)

The vector size, or dimensionality (embedding_length: xxx), is the number of numbers in each embedding vector. Common embedding models produce vectors ranging from 384 dimensions (e.g., all-MiniLM-L6-v2) to 3072 dimensions (text-embedding-3-large). Higher dimensions capture more semantic details but require more storage and computational resources for database indexing and search. Some models allow you to shorten vectors (e.g., use only 256 out of 3072 dimensions) to save space while retaining high performance. Vector count refers to the total number of vectors stored, which usually corresponds to the number of content chunks indexed +the overlap chracters space. More vectors mean more granularity for search and retrieval but also increase database size and operational overhead sometimes 5times the size and also need more time for response. Chunk Length Chunk length is the size (usually measured in words, tokens, or characters) of the text split for embedding (ALLM chunk length/ chunk size -> in characters). here is a tokenizer calculator and a Vram calculator - (you need the original model link NOT the GGUF) second VRAM calc for GGUF -> YOU need format "https://huggingface.co/provider/model/blob/main/model.gguf" Example: "https://huggingface.co/unsloth/granite-3.3-8b-instruct-GGUF/blob/main/granite-3.3-8b-instruct-UD-Q8_K_XL.gguf"

...

How embedding and search works:

Embeddding is not a database, you can not ask how many times the word "xyz" appears. With embedding, you cannot ask for a summary of an entire book. Example: You have a txt/pdf file maybe 90000words(~300pages, ~1'000'000chars) a book. You ask the model lets say "what is described in chapter called XYZ in relation to person ZYX". Now it searches for keywords or similar semantic terms in the document. if it has found them, lets say word and meaning around “XYZ and ZYX” , now a piece of text 2000chars around this word “XYZ/ZYX” is cut out at this point. (In reality, it's all done with coded numbers per chunck and thats why you dont can search for single numbers or words, but dosnt matter - the principle) This text snippet is then used for your answer. ...

Nevertheless, the main model is also important !

Especially to deal with the context length and I don't mean just the theoretical number you can set. Some models can handle 128k or 1M tokens, but even with 16k or 32k input the response with the same snippets as input is worse than with other well developed models. Try to find a model that match your language! Instruct models are better to answer questions from document! For 8000 token and more context use models 8b better more. If you are trying to tell stories from the answer, use other models...

...