Coding Test for Llama 3: Implementing JSON Persistence
We put Meta's new Llama 3 large language model through its paces as a coding tool by asking it to create a JSON persistence solution.
Apr 27th, 2024 5:00am by
Image via Unsplash+.
When looking at a previous LLM (Llama 2), I asked it to do some long multiplication that it failed to manage. It did however “fail like a human.” Let us see how this model has progressed:
A quick, and much better-looking answer. Still wrong; the answer is 1,223,834,880. However, we aren’t here to worry about this known lacuna. Let’s return to the main topic: persistence with JSON.
Saving with JSON
JSON is a portable data format, and while less efficient than an arbitrary native binary format, it can at least be checked by humans and implemented anywhere. Because JSON can handle basic value types and some collections, we can in general map our instance data to JSON format. This is known as marshalling and is a lot less of an issue today than it used to be. Whether you want to persist (i.e., save and load) user preferences, a turn in a game or just some plain old useful data, JSON has your back. In this post I’ll use .NET and C#. While we can do just fine with the inbuilt System.Text.Json, the Newtonsoft package is a bit nicer, so I’ll call that. First I’ll set up a console project that will use the new style top-level statements to cut out the need for main statements and other boilerplate. So in a terminal:
I started with a quick Car class, which I wrote directly in Program.cs. We will use this as our target object:
Car car = new Car("Saab", "V90", Car.CarColor.Red);
Console.WriteLine($"First car: {car}");
public class Car
{
public enum CarColor {Red, Blue, White};
public string Brand;
public string Model;
public short MileageAtLastService;
public Car(string brand, string model, CarColor color)
{
this.Brand = brand;
this.Model = model;
this.MileageAtLastService = 0;
this.Color = color;
}
public void SetMileage(short mileage) => this.MileageAtLastService = mileage;
public override string ToString() => $"{Color} {Brand} {Model}" ;
}
First car: Red Saab V90
JsonMarshalling> dotnet add package Newtonsoft.Json
using CarClass;
using Newtonsoft.Json;
const string OUTFILE = @".cars.json";
Car car = new Car("Saab", "V90", Car.CarColor.Red);
Console.WriteLine($"First car: {car}");
string output = JsonConvert.SerializeObject(car);
Console.WriteLine($"Serialised: {output}");
Serialised: {"Brand":"Saab","Model":"V90","MileageAtLastService":0,"Color":0}
using Newtonsoft.Json
using Newtonsoft.Json.Converters;
...
[JsonConverter(typeof(StringEnumConverter))]
public CarColor Color {get; set;}
...
Serialised: {"Brand":"Saab","Model":"V90","MileageAtLastService":0,"Color":"Red"}
using CarClass;
using Newtonsoft.Json;
const string OUTFILE = @"./cars.json";
List<Car> cars = new List<Car>();
Car car = new Car("Saab", "V90", Car.CarColor.Red);
Console.WriteLine($"First car: {car}");
cars.Add(car);
car = new Car("Volkswagen", "Polo", Car.CarColor.White);
Console.WriteLine($"Uncle's car: {car}");
cars.Add(car);
string output = JsonConvert.SerializeObject(cars);
File.WriteAllText(OUTFILE, output);
Let’s generalize the persistence part and put it in its own project file:
using CarClass;
using Newtonsoft.Json;
namespace PersistenceServices
{
public static class Persistence
{
const string JSONFILE = @"./cars.json";
public static void WriteCarsToFile(List<Car> cars)
{
WriteToFile(cars);
}
public static List<Car> ReadCarsFromFile()
{
return ReadFromFile<Car>(JSONFILE);
}
private static void WriteToFile<T>(List<T> list)
{
string jsonString = JsonConvert.SerializeObject(list, Formatting.Indented);
File.WriteAllText(JSONFILE, jsonString);
}
private static List<T> ReadFromFile<T>(string filename)
{
List<T>? list = null;
if (File.Exists(filename))
{
string jsonString = File.ReadAllText(filename);
list = JsonConvert.DeserializeObject<List<T>>(jsonString);
}
else Console.WriteLine("No existing file found for " + filename);
return list ?? new List<T>();
}
}
}
using CarClass;
using Newtonsoft.Json;
using PersistenceServices;
List<Car> cars = new List<Car>();
Car car = new Car("Saab", "V90", Car.CarColor.Red);
Console.WriteLine($"First car: {car}");
cars.Add(car);
car = new Car("Volkswagen", "Polo", Car.CarColor.White);
Console.WriteLine($"Uncle's car: {car}"); cars.Add(car);
Persistence.WriteCarsToFile(cars);
List<Car> cars2 = Persistence.ReadCarsFromFile();
cars2.Last().MileageAtLastService = 2500;
Persistence.WriteCarsToFile(cars2);
OK, so we created some persistence to a simple file using JSON, using a specific package. We ended up with these files in our project workspace:
Now let’s try to see how far we can get with Llama 3. Let’s break down what we did.
- Created a simple class to represent an owned car.
- Turned it into json.
- Used the json format to save and load into a local file.
Great. It also gave examples of using the class and setting the properties. Let us see if we can persuade it to make the color property an enum:
It did! Even with my spelling mistake in the request!
Now let’s see if it can use JSON to convert. Let’s force it to use Newtonsoft.json too.
It thinks this will be the output:
Of course, we already know this won’t work as it is. We need to fix the enum. We could try to tell Llama this, but we only know it isn’t correct because we’ve been here before. However, let’s take the win.
The good news is that Llama 3 was quick, well formatted and good with basic requirements.
- It allowed us to work on the code in a continuous fashion over time in one thread.
- It understood how to use an arbitrary named package.
- It correctly created example instances of Car, as well as some nice car colors.
- The answers are nicely formatted.
YOUTUBE.COM/THENEWSTACK
Tech moves fast, don't miss an episode. Subscribe to our YouTube
channel to stream all our podcasts, interviews, demos, and more.
