Update README.md

README.md

@@ -81,32 +81,41 @@ A combination of the [SmoothQuant](https://arxiv.org/abs/2211.10438) and [GPTQ](
 
 ## Deployment
 
-This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
+1. Initialize vLLM server:
+```
+vllm serve RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w8a8 --tensor_parallel_size 1 --tokenizer_mode mistral
+```
+
+2. Send requests to the server:
 
 ```python
-from vllm import LLM, SamplingParams
-from transformers import AutoProcessor
+from openai import OpenAI
 
-model_id = "RedHatAI/Mistral-Small-24B-Instruct-2501-FP8-quantized.w8a8"
-number_gpus = 1
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://<your-server-host>:8000/v1"
 
-sampling_params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256)
-processor = AutoProcessor.from_pretrained(model_id)
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
 
-messages = [{"role": "user", "content": "Give me a short introduction to large language model."}]
+model = "RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16"
 
-prompts = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
 
-llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
+messages = [
+    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
+]
 
-outputs = llm.generate(prompts, sampling_params)
+outputs = client.chat.completions.create(
+    model=model,
+    messages=messages,
+)
 
-generated_text = outputs[0].outputs[0].text
+generated_text = outputs.choices[0].message.content
 print(generated_text)
 ```
 
-vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
-
 <details>
   <summary>Deploy on <strong>Red Hat AI Inference Server</strong></summary>