How to Generate Software Release Notes with LLMs and a Memory Pack

Release notes are an important piece of information for communicating with the customers about the recent changes of your product. It allows users to understand the new features of the products, the major improvements of existing features, bug-fixes, and other updates that they care about. Not only does it keep users informed about what’s happening in the product, it also provides a historical record of the changes, helps users to take advantage of changes, and more. But writing release notes can take tens of hours of work easily.

Why is it difficult to write release notes?

There are several reasons:

1. Target Audience Changes. Different groups of people have different levels of understanding of the product. When the engineering team implements new features and fixes bugs of the product, they may use vocabulary that is too technical for the audience. It would become technical jargon or information overload for users.
   
   
2. Specification and Implementation Mismatch. The implementation of the solution may not be completely aligned with the initial description made in the technical specification. Oftentimes, people communicate in various ways, and decide to adjust the solution based on the discussions. These changes are not necessarily recorded in the project management tool. However, they impact the final delivery of the product and it is time-consuming to collect this kind of information.
   
   
3. Categorization. The changes can be categorized in different types, each of them can be further categorized into sub categories. It depends on the complexity of the product, the release cycles, the information that you want to share in the release notes, and the scope of the release. The change items can be grouped by applications, components, the importance of the changes, layers in the technical stack (libraries, applications, infrastructure, …), etc. It requires a good understanding of similar vocabularies, making the relationships between the code changes and the product, and many more.
   
   
4. Multiple Data Sources. Sometimes, the release notes cannot be collected by one single source. You may want additional information from different sources to make the release notes more compelling. You may need tickets from your project management, RFCs from your engineering team, comments in the code-review, and some other sources to gather information.
   
   
5. Review Process. The review process can be slow because it requires back-and-forths between you and the software engineers, project managers, product owners, or other roles to sync and validate the document before publishing it. It is important to keep track of the source of the information to serve as references for the corrections.
   
   
6. Translation. Sometimes, you also want to translate the release notes into multiple languages so that it can fit readers from different countries.

How do we create release notes with LLMs and a Memory Pack?

At Vertesia, we create release notes with the help of LLM. The creation of our release notes is divided into two parts: data collection, release notes generation, and manual review. In the first part, data collection, we collect all kinds of information related to the release notes to prepare for the generation. And in the second part, we generate the actual release notes and translate them using a large-language-model (LLM). And finally, we let our release manager review the changes and perform manual adjustments before publishing the release notes.

Data Collection and Memory Pack

We collect different kinds of data from the project management tool (GitHub Issues) and from our version control system (Git and GitHub pull-requests). We put all kinds of information into one single place for content generation: the Memory Pack. Memory Pack is an image that packages the context to be used when expanding LLM prompts. It allows users to decouple the data collection of the adjustment and the execution, and it serves as an immutable context for the LLM.

Using a Memory Pack has multiple advantages. First of all, it allows us to decouple data collection from the data generation, it serves as a boundary between two phrases. Oftentimes, the release notes generation requires improvements, but we were not sure whether they refer to the prompts or they refer to the quality of the data that we provided as inputs. Having the Memory Pack allows us to easily inspect the content of the data. If the content of some file is empty, it is obvious that the problem happened during the data collection process. Secondly, it allows us to store an important amount of data for the prompts as the context for the LLM. Typically, if you want to provide a list of issues and the code difference as part of the prompt, it could be hard to do so without a Memory Pack. Most of the online editors only support small chunks of data. Finally, using Memory Pack allows you to create a small data collection pipeline as a Recipe using Memory Commands in Typescript. We will discuss more in detail later in this post.

Now, let’s take a look at the file structure of a Memory Pack. If we extract the tarball, we will see different sources of data, represented as a file or a directory. In this example, we can see that GitHub Issues are grouped into a directory, GitHub pull-requests are grouped into another directory, and all the commits and the code differences are stored in a single file. This is the structure we had chosen for our team, but you are free to adapt the structure to fit your needs. The only mandatory file is metadata.json, which contains the properties served as the input arguments for the Memory Pack. They are useful for the prompts.

Recipe

Now you understand the basics of the Memory Pack, let’s take a look into the Recipe. Recipe is a text document written in Typescript, that contains all the commands a user could call on the command line to assemble a Memory Pack. The Vertesia CLI can build Memory Pack automatically by reading the instructions from a Recipe. Here are some key instructions that could be used when collecting information for release notes:

You can see the full list of instructions on our GitHub repository.

A Recipe can be used for building a Memory Pack via Vertesia CLI. When doing so, the CLI will execute the instructions inside the script “release-notes.ts”, and upload the final result to Google Cloud Storage. This command assumes that you have a project registered on our platform.

Release Notes Generation

The release notes generation is then split into two parts: the generation and the translation.

For the generation part, you can rely on the Interaction feature of the Vertesia Studio. An Interaction defines the tasks that the LLM is requested to perform. Inside the Interaction for release notes generation, you can specify the instructions for the LLM as one or multiple prompts. Typically, you can specify the target audience, the structure of the release notes, how to categorize the items, the format of each item, etc. When running an Interaction, you can specify the Memory Pack so that you can inject some pieces of values of the Memory Pack to the Interaction. You can also switch the LLM model and decide which one is the best in terms of performance and cost. You can also fine-tune the LLM options to adjust the result, such as adjusting the temperature to control the hallucinations. To integrate this Interaction into your system, you can rely on the Vertesia command line to call the Interaction. Here is example:

For the translation part, you can rely on another Interaction to translate the content for you. Compared to using another translation service, using Vertesia allows you to group both needs together and simplify the translation. You can also use another LLM, less perform then the previous one, since translating the content is relatively easier then the synthesizing of all the changes between two releases.

If we put all the pieces together, here is a diagram summarizing the key steps of the release notes generation. In the business layer (yellow), you can see the processes for generating the release notes, in the application layer (blue), you can see how those actions can be achieved using the Vertesia Platform. And in the technology layer (green), you can see what are the technologies used for supporting the generation. 

Benefits

Why should you use Vertesia to generate release notes? Here are some benefits from what I see:

• Support multiple data sources. With the Memory Pack and the Recipe, it’s easy to build your data collection pipeline with a simple script. You can invoke other command line tools to collect information, to fit organization needs. You can also decide the data format without having to follow constraints imposed by the platform.
  
  
• Adjust content to target audience. Compared to traditional release notes generation tools, such as GitHub built-in release note generator, it can take into account much more parameters for the generation, which improve the accuracy of the information and maintain the interests of the audience.
  
  
• Improve categorization. Compared to a label-based or tag-based system, leveraging LLM makes the categorization even better. You can categorize changes in other ways, directly in the prompt.
  
  
• Facilitate review process. Using the LLM for release notes generation may require a lot of back-and-forth to adjust to content until everyone is satisfied with the results. Relying on the Memory Pack allows you to easily identify which part of the process needs to be improved and therefore improve your efficiency.

Conclusion

In this blog, we shared how we can generate release notes easily with Vertesia. We introduced the new concept of the Recipe (Memory Commands) and the Memory Pack, which provides a portable solution for data collection and data storage. Then we saw how to use Studio to write a prompt and run the prompt inside an Interaction to interact with a LLM. Finally, we showed the whole solution and shared some benefits of choosing Vertesia for this kind of task.

To see Vertesia in action, schedule a live demo with one of our LLM experts.