Chatbot 3: Long Term Memory using a file

In the previous post, we covered Golang Slices used for short term memory for the chatbot. As we noted – this only lasts as long as the program is running. Now, let’s see if we can keep information when the chatbot goes offline. That is – long term memory. There are a few different approaches for long term memory. Writing to a file or to a database are the most common. For this example, we’re going to use a file. For future development, most use a database. This allows the database to handle the i/o (input/output) and maintain performance. This is because a database is designed specifically for the use case of storing and retrieving data.

Step 1: Breaking it down

Conceptually, File I/O is actually much more straightforward then previous concepts. The program needs a destination file, a payload, and permission when writing to a file. It needs a source file, destination and permission when reading from a file. It will also need to know how to interpret or parse the data when reading. Thus, it also needs to store the payload in a consistent format. There are a large number of guides in any given programming language for writing to / reading from a file. I’m not going to recommend one currently – and will focus on the actual implementation.

Step 2: File I/O in golang

The first example we have, is creating a file. Here is how to create ‘test.txt’ and avoid a memory leak. A memory leak is when something is reference by the program, but the code flow of the program no longer has context for that reference.

// Get a read/write file reference by creating it if it doesn't exist, or appending to it if it does exist.
file, err := os.OpenFile("test.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666) // This provides a reference to the file 'f', or an error if it cannot. 

// check for the error. 
if err != nil { // a 'nil' error means no error occurred. 
  fmt.Println(err) // print the error
  return // stop processing the context.
}

// ##IMPORTANT## Always close the file reference to prevent a memory leak!
defer func() { // defer is an instruction that runs when the 'context' closes.
	_ = file.Close() // close the reference!, the underscore is a way to ignore the value returned by Close()
}()

Reading from an open file reference

// Reading each line in a file into an array
var lines []string // declare the slice to hold the data
scanner := bufio.NewScanner(file) // use the buffer input output package to start a file scanner
for scanner.Scan() { // loop through the file scanner
  lines = append(lines, scanner.Text()) // add the scanned line to the lines slice.
}

Writing to an open file reference

text := "Coming to a file near you" // string to write to file
_, err = fmt.Fprintln(file, text) // write the data to the file
if err != nil { // error handling if file I/O fails.
   fmt.Println(err)
}

\\ or alternatively
linesWritten, err := f.WriteString(text)

Step 3: Date storage format

There are many ways data can be stored within a file. If you’ve messed with files in the past – you’ll know a bit about text .txt files. Maybe you’ve seen markdown .md files. There are many different types of files – and these all serve different purposes. markdown, text, and even .csv files are all Human Readable Formats. If you open any of these in a text editor, you’ll be able to make sense of them quickly. However – they aren’t exactly the best for storing data from a program. For this particular example, we are going to simply use a txt file to place human readable data into the file.

Step 4: Chatbot Enhanced Memory

1. launch program
2. check if history exists (create it if it doesn’t exist)
   1. load the history into the history slice
3. send a friendly greeting and store it to history
4. send a simple instruction and store it to history
5. Have the user put something in and store it to history
6. echo what the user put in and store it to history
7. Check if the user typed ‘history’
   1. Store the request to history
   2. don’t repeat the history in the history file, just count the number of lines.
   3. Loop through the history slice
   4. print the key and the value
8. repeat 4-7 until the user says ‘bye’

Step 5: Putting it all together

Now that we’ve covered the details, we can run the program. We’ll find that it isn’t much different from running the program during lesson 2. However – when we say ‘bye’ and start the program again, that’s where the changes are noticeable.

1. run the program
2. type hello
3. type history
4. type bye
5. run the program again
6. type history

Here is the final example used above. It is also available in github in chatbot, lesson 3

func main() {
	f, err := os.OpenFile("history.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)

	// check for the error.
	if err != nil { // a 'nil' error means no error occurred.
		fmt.Println(err) // print the error
		return           // stop processing the context.
	}

	defer func() {
		_ = f.Close()
	}()

	// string for storing input
	var input string

	// Slice for storing the history of the world.
	var history []string
	scanner := bufio.NewScanner(f) // use the buffer input output package to start a file scanner
	for scanner.Scan() {           // loop through the file scanner
		history = append(history, scanner.Text()) // add the scanned line to the lines slice.
	}

	fmt.Println("Hello, World!")
	fmt.Println("I am Echo! Please tell me something to say by typing it in, and pressing enter!")
	_, _ = f.WriteString("Hello, World! \n")
	_, _ = f.WriteString("I am Echo! Please tell me something to say by typing it in, and pressing enter! \n")
	// Stop when we see "bye"
	for input != "bye" {

		fmt.Print("You: ")
		scanner := bufio.NewScanner(os.Stdin)
		for scanner.Scan() {
			input = scanner.Text()
			break
		}
		_, _ = f.WriteString("You: " + input + "\n")
		// Add the new input to the history
		history = append(history, input)
		fmt.Print("Echo: " + input + "\n")
		_, _ = f.WriteString("Echo: " + input + "\n")
		// Check if the input was the string 'history'
		if input == "history" {
			// We won't add the history to the history, instead we can add how many lines of history exist.
			_, _ = f.WriteString("<Truncated> " + strconv.Itoa(len(history)) + " Lines of history repetition.")
			// iterate (loop) through the history
			for k, v := range history {
				// print out the Key and the Value
				fmt.Println(k, v)
			}
		}
	}
}

I don’t know about you – but that code is starting to look like my pantry after my kids try finding the peanut butter. In our next lesson – we will start to look at organizing code a little bit better.