Inferencing a Mistral-7B-Instruct Model

Mistral-7B

Mistral 7B is an Large Language Model (LLM) designed for high performance and efficiency. It uses grouped-query attention (GQA) for faster inference and sliding window attention (SWA) to successfully handle sequences of any length with a lower inference cost.

Mistral 7B — Instruct, model fine-tuned to follow instructions on instruction datasets.

There are 2 versions of instruct models available so far Mistral-7B-Instruct-v0.1 and Mistral-7B-Instruct-v0.2.

Lets se how to inference this model, before that ensure that following python packages are in your system

Installing transformers from source should solve the issue

pip3 install git+https://github.com/huggingface/transformers

def mistral_v1():
   model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1",device_map="auto")
   tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
   while True:
      query = input("Enter your prompt: ")
      messages = [
         {"role": "user", "content": str(query)}
      ]
      encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
      generated_ids = model.generate(encodeds, max_new_tokens=300, do_sample=False)
      decoded = tokenizer.batch_decode(generated_ids)
      print("Query: ",query)
      print("Result")
      print(decoded[0])

Similarly we can do v0.2 also

def mistral_v2():
   model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2",device_map="auto")
   tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
   while True:
      query = input("Enter your prompt: ")
      messages = [
         {"role": "user", "content": str(query)}
      ]
      encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
      generated_ids = model.generate(encodeds, max_new_tokens=300, do_sample=False)
      decoded = tokenizer.batch_decode(generated_ids)
      print("Query: ",query)
      print("Result")
      print(decoded[0])

Example output

Enter your prompt: write a biography about Nicola Tesla

   Result
   <s> [INST] write a biography about Nicola Tesla [/INST] Title: "A Serbian Dream: The Extraordinary Life of Nikola Tesla"

   Introduction:
   Nikola Tesla (1856-1943), a Serbian-American inventor, electrical engineer, mechanical engineer, and futurist, is best known for his contributions to the design of the modern alternating current (AC) electricity supply system. Born in Smiljan, Croatia, then part of the Austro-Hungarian Empire, Tesla's groundbreaking inventions and discoveries in the field of electricity would change the world and pave the way for the technological advancements of the 20th century.

   Early Life and Education:
   Born on July 10, 1856, Tesla was the fourth child of Serbian Orthodox priests Serafim and Đokica Tesla. He showed an early aptitude for mathematics and science, and his natural curiosity led him to experiment with electromagnets and other electrical devices. In 1873, Tesla left home to pursue an education in engineering in Austria and the Czech Republic. He studied at the Austrian Polytechnic in Graz and the Charles-Ferdinand University in Prague, where he focused on physics and mathematics.

   Career and Inventions:
   In 1884, Tesla emigrated to the United States

Try it yourself 🚀🌟