Gemma: Google’s new open LLM and Inferencing Gemma

Google Gemma

Gemma is a family of 4 new LLM models by Google based on Gemini. It comes in two sizes: 2B and 7B parameters, each with base (pretrained) and instruction-tuned versions. All the variants can be run on various types of consumer hardware, even without quantization, and have a context length of 8K tokens:

• gemma-7b: Base 7B model.
• gemma-7b-it: Instruction fine-tuned version of the base 7B model.
• gemma-2b: Base 2B model.
• gemma-2b-it: Instruction fine-tuned version of the base 2B model.

Gemma comes in two sizes: 7B parameters, for efficient deployment and development on consumer-size GPU and TPU and 2B versions for CPU and on-device applications. Both come in base and instruction-tuned variants.

Working with Gemma

To use Gemma models with transformers, make sure to use the latest transformers release:

pip3 install -U "transformers==4.38.1" --upgrade

The following snippet shows how to use gemma-7b-it with transformers. Its validated using aws p3.8xlarge instance. Here i tried both gemma-7b and gemma-7b-it (instruction) models.

def google_gemma():
    # tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
    # model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto")
    # inference time
    # 1. 4.50 mins
    # 2. 5.10 mins

    # instruction model
    tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b-it")
    model = AutoModelForCausalLM.from_pretrained("google/gemma-7b-it", device_map="auto")
    # inference time
    # 1. 1.42 mins
    # 2. 1.16 mins

    while True:
        query = input("Enter your prompt: ")
        input_ids = tokenizer(query, return_tensors="pt").to(device)
        outputs = model.generate(**input_ids,max_new_tokens=300, do_sample=False)
        print("Query: ",query)
        print("Result")
        print(tokenizer.decode(outputs[0]))

Sample outputs

Enter your prompt: write a biography about Oommen Chandy


Query:  write a biography about Oommen Chandy
Result
<bos>write a biography about Oommen Chandy

    Oommen Chandy (born 1948) is a former Chief Minister of Kerala, India. Born into a Syrian Christian family, he was born in the village of Thiruvananthapuram and completed his education at St. Xavier's College, Palayamkottai. He began his political career in the Indian National Congress and rose through the ranks to become the Chief Minister in 1996.

    Chandy's tenure as Chief Minister was marked by both controversy and achievement. He is credited with introducing the concept of "People's Pension Scheme" and the "Keraala Model" of poverty alleviation. However, he also faced criticism for his handling of the 2002 Kerala floods and the 2008 Kerala agrarian crisis.

    Chandy's political career ended in 2016 when he resigned from the Congress due to disagreements over the party's leadership. He has since formed his own political party, the Kerala Congress Party.

    Oommen Chandy is a controversial figure in Indian politics. He is known for his strong leadership and his ability to bring about change. However, he is also known for his controversies and his sometimes erratic decision-making. Despite his flaws, Chandy remains a prominent figure in Kerala politics and continues to advocate for the people of his state.<eos>

Enter your prompt: Write a paragraph about Issacc Newton

    Query:  Write a paragraph about Issacc Newton
    Result
    <bos>Write a paragraph about Issacc Newton's contributions to mathematics and physics.

    Sir Isaac Newton, born in 1642, was a prominent English mathematician and physicist whose groundbreaking contributions revolutionized the fields of mathematics and physics. Newton's most significant achievements include the development of the concept of motion and inertia, the law of gravitation, and the theory of universal gravitation. He revolutionized calculus with his invention of the differential and integral methods, which paved the way for advancements in mathematics and physics. Newton's contributions to physics include his theories on motion, equilibrium, and motion of fluids, which profoundly influenced scientific understanding. His contributions to mathematics include his development of the binomial coefficient formula and his pioneering work on infinite series. Newton's legacy as a leading scientist continues to inspire and shape scientific thought and understanding.<eos>

Try it yourself 🤗