A Fluid Truck engineer’s elegant solution

As a rapidly growing tech company, Fluid Truck devs are constantly having to find solutions to novel problems. One example of this is Samantha MacIlwaine’s invention of DeepCopy. With DeepCopy, Fluid Truck engineers and developers everywhere can make their programming more efficient.

At Fluid Truck we program with Golang, or “Go,” which is a programming language designed at Google. It’s extremely popular, and is focused on simplicity and efficiency.

If you’re not well-versed in programming, allow us to explain DeepCopy in simple terms. Basically, Fluid Truck initially had one folder for code. Then, the team decided to split it up into multiple folders to make scaling easier. However, each folder needs to have a models file where data structures are defined.

Rather than manually recreating these models files and translating them between folders, which is repetitive and time-consuming, Sam invented a function called DeepCopy. This program is used to replace all the bulky translation functions with just 1 line, making the model translation much easier.

Sam’s invention isn’t just big news for Fluid Truck. DeepCopy helps web developers everywhere scale quickly and efficiently. It’s why Sam made it open source and free to the public. Now everyone can use Sam’s invention to save time while programming.

If you are a brilliant programmer like Sam, keep reading for the nitty gritty details on this impressive new tool.

Background

A few months ago, the backend engineering team at Fluid restructured our Golang codebase to make it more scalable. We broke up the existing monorepo into many microservices and set them up to communicate via gRPC.

Fluid engineers work on many different products that are often very specialized, and a microservice architecture allows us to independently deploy smaller, highly testable pieces of code without the risk of interrupting another team’s progress.

While this approach provided many benefits, especially as the team expanded, one drawback we faced was dealing with shared data models across different projects.

An example of a shared data model is shown below.

type Vehicle struct {
	Id uint
  Make string
  Model string
  Color string
  AddedAt *time.Time
}

Multiple microservices often need to use the same data model. For example, our Vehicle Maintenance microservice and our Reservations microservice might need to both exchange data in the shape of the Vehicle struct. So how do we handle this?

The idea of importing models from other microservices is unattractive because it would destroy the microservices’ independence from each other.

But similarly, writing the data model into every individual microservice would require an excessive amount of repeated code and would make it difficult to keep all data models in sync.

The best option was to create a shared library that all microservices can import, with versioning control to keep track of changes. We called it Service Clients Internal, or SCI.

We decided to implement this with a protocol buffer compiler that generated Golang models for our library. Every time a model was changed or added, a new version of the SCI library would be released, and each microservice could upgrade to the new version when it was ready.

However, each microservice still needed local data models to use in conjunction with database operations. Thus, we faced the problem of converting between the shared SCI models and each microservice’s local models, which often differed in small but significant ways.

The problem

For example, let’s say we are sending a request to update some attribute of the vehicle. The update request object might look like this:

type Vehicle struct {
	Id uint
	Make *string
	Model *string
	Color *string
	AddedAt *timestamppb.Timestamp
}

You can see that there are 2 main differences between this model and the previous Vehicle model. First, the AddedAt field is of type *timestamppb.Timestamp. This is because our protocol buffer compiler, Google’s protobuf package, used the timestamppb.Timestamp object to store timestamps.

Second, the values for Make, Model, and Color are pointers so that we can tell which attributes should be left alone versus which attributes should be changed. For example, if we only wanted to change the vehicle’s color, we would send in this struct:

  req := Vehicle{Color: proto.String("blue")}
  resp, err := server.UpdateVehicle(req)

It is useful for us to be able to designate certain fields as required or optional based on whether or not they are pointers. However, the optional fields in a grpc request object often do not match the optional fields in a local model.

In this example, every single Vehicle in our database will have a make, model, and color. But in the request to update a Vehicle, all 3 of those attributes are optional and each may not exist.

Since the local model is the one that loads information from the database, we need to be able to convert data between our SCI library Vehicle model and our local Vehicle model.

Initial solution: Custom conversion functions

The brute-force way to handle data conversions between SCI and local models was to create custom functions for every type of conversion.

(Note: We also had reusable conversion functions, called toTimestampPointer and toTimePointer, between timestamp.Timestamp pointers and time.Time pointers.)

func toSCI(lv *local.Vehicle) *sci.Vehicle {
	return sci.Vehicle{
		Id: lv.Id,
		Make: &lv.Make,
		Model: &lv.Model,
		Color: &lv.Color,
		AddedAt: toTimestampPointer(lv.AddedAt)
	}
}

func fromSCI(v *sci.Vehicle) *local.Vehicle {
	lv := &local.Vehicle{
		Id: v.Id,
		AddedAt: toTimePointer(v.AddedAt),
	}
	if v.Make != nil {
		lv.Make = *v.Make
	}
if v.Model != nil {
	lv.Model = *v.Model
}
if v.Color != nil {
	lv.Color = *v.Color
}
	return lv
}

Our gRPC handler functions would look something like this:

func (h *handler) handleUpdateVehicle(req *sci.Vehicle) (resp *sci.Vehicle, err error) {

	// convert from SCI to local model
	vehicleToUpdate := fromSCI(req)

	// perform operation
	updatedVehicle, err := h.controller.UpdateVehicle(vehicleToUpdate)
	if err != nil {
		return nil, err
	}

	// convert from local model back to SCI
	resp = toSCI(updatedVehicle)
	
	// return
	return resp, nil
}

As you can imagine, writing two extra functions for every single new grpc model was extremely inefficient. If you include unit tests, we wrote an average of 400 extra lines per grpc call, just for these conversion functions. Yikes!

If we could somehow make these conversions automatic, then we could save an enormous amount of development time.

Research: Existing open source conversion functions

There are several open source “deep copying” libraries that we attempted to use for model conversion.

While all of these options offered nested conversion between similar structs, which was great, none of them offered the ability to convert between infinite layers of non-pointers and pointers, nor to convert between timestamppb.Timestamp and time.Time objects. Therefore, we could not use them as the universal, one-size-fits-all solution that we wanted.

During our department-wide Engineering Hack Day in November 2021, we decided to try and create our own model conversion function. Ideally, it would be able to recursively copy over the content of one object into the format of another, even if the two objects were unequal.

After completing the project, we would be able to convert between any two struct fields that could be theoretically cast to each other, such as int32 and string, or even ****string and *float64, in either direction.

Introducing DeepCopy

We succeeded in creating a single, universal function that converts any model into any other model, throwing an error only if the underlying fields are strictly incompatible, such as uint64 and bool.

DeepCopy is public and freely available to use under the BSD-3-Clause License. Check it out here.

Now that we have DeepCopy, converting data between the two structs is an easy, 1-line solution. Let’s go back to our two models.

// local model
type Vehicle struct {
	Id uint
	Make string
	Model string
	Color string
	AddedAt *time.Time
}

// SCI model
type Vehicle struct {
	Id uint
	Make *string
	Model *string
	Color *string
	AddedAt *timestamppb.Timestamp
}

Let lv equal a local Vehicle object, and v equal an SCI Vehicle object.

err := deepcopy.DeepCopy(v, &lv)

We can also convert in the opposite direction.

err := deepcopy.DeepCopy(lv, &v)

Benefits of DeepCopy

This kind of one-line conversion, between models with unequal fields that include a mix of pointers and non-pointers, has never been possible before in Go.

But in addition to converting nested objects, DeepCopy can also be used as a lazy typecasting tool for all kinds of objects, even primitive types.

  x := uint64(5)
  y := ""
  err := deepcopy.DeepCopy(x, &y)
  print(y) // prints "5"

It can also be used to create true deep copies of nested objects.

type StructA struct {
	CreatedAt time.Time
}
type StructB struct {
	FieldOne string
	FieldTwo *StructA
}
myStruct := &StructB{
	FieldOne: "hi",
	FieldTwo: &StructA{
		CreatedAt: time.Now()
	}
}
myStructCopy := &StructB{}
err := deepcopy.DeepCopy(myStruct, myStructCopy)

Now, myStructCopy is a true, separate copy of myStruct.

Limitations of DeepCopy

One limitation for DeepCopy is that the second argument always needs to be a pointer, due to the Laws of Reflection regarding settability. Can you spot the error below?

  x := "hi"
  y := ""
  err := deepcopy.DeepCopy(x, y)

To fix it, simply call the address of y.

  err := deepcopy.DeepCopy(x, &y)

Perfect!

Another limitation is that DeepCopy does not currently support mapping between two sets of enumerated fields.

Conclusion

DeepCopy supports our microservice architecture and improves scalability by allowing our team to make easy, 1-line conversions between different data models in Golang.

DeepCopy is open source and free to the public! Check out the code and documentation at https://github.com/fluidtruck/deepcopy.