In this blog post, I am going to show you how I tested my HTTP API in Go.

The endpoints

Before jumping to the test code, I want to show you the two endpoints that I am testing

  • POST /users to create user
  • GET /users to list users

You can go to the full source code by clicking here, but I am going to take you through the code snippet here as well. Feel free to skip to the next section if you are comfortable with the code.

Like many Go applications, the starting point for this web app is the main function. It listens to port 3000 and handles the 2 endpoints mentioned above. Along with the main function, in api.go, I also have userStore that will serve as a storage when a new user is created and userIdCounter that will increment when a new user is successfully created.

//Inside main.go
func main() {
    fmt.Println("Server starting")
    http.ListenAndServe(":3000", api.Handlers())
}
//Inside api.go
var userIdCounter uint32 = 0

var userStore = []User{}

type User struct {
    Id           uint32 `json:"id"`
    Username     string `json:"username"`
    MoneyBalance uint32 `json:"balance"`
}

func Handlers() *mux.Router{
    r := mux.NewRouter()

    r.HandleFunc("/users", createUserHandler).Methods("POST")

    r.HandleFunc("/users", listUsersHandler).Methods("GET")

    return r
}

I am using Gorilla Mux to route my request. If you have never seen this package before, definitely check it out. It has tools that will make your life easier when building web application.

Next up is the POST /users endpoint for creating user. The handler function for this endpoint is createUserHandler. In it, I am calling validateUniqueness to ensure that username is unique across all users. If validation passes, I create a new user and increments the ID counter.

func createUserHandler(w http.ResponseWriter, r *http.Request) {
    p := UserParams{}

    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Printf("Error: %s\n", err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    err = json.Unmarshal(body, &p)
    if err != nil {
        fmt.Printf("Error: %s\n", err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    err = validateUniqueness(p.Username)

    if err != nil {
        fmt.Printf("Error: %s\n", err)
        w.WriteHeader(http.StatusBadRequest)
        return
    }

    u := User{
        Id:           userIdCounter,
        Username:     p.Username,
        MoneyBalance: p.MoneyBalance,
    }

    userStore = append(userStore, u)

    userIdCounter += 1

    w.WriteHeader(http.StatusCreated)
}

func validateUniqueness(username string) error {
    for _, u := range userStore {
        if u.Username == username {
            return errors.New("Username is already used")
        }
    }

    return nil
}

The last endpoint is GET /users which lists all the users in userStore.

func listUsersHandler(w http.ResponseWriter, r *http.Request) {
    users, err := json.Marshal(userStore)

    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    w.Write(users)
}

If you want to run this web app, please checkout the github README.

The tests

There are several cases that I want to test here.

  • Successful user creation
  • Error when creating user witn non-unique username
  • Successful listing of user records in JSON

Note that this is not comprehensive. I would definitely take more time to consider other cases that might break your endpoints e.g. broken JSON.

Setting up the server

You can think of testing HTTP endpoints in Go as starting the HTTP server, sending request to the endpoint and making some assertions about the response. This is what we are going to do.

Under the api package, you want to create a new file called api_test.go. In it, create an init function that will create a new HTTP server for testing and get the user URL. Remember to import all the necessary packages.

package api_test

import (
    "api"
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
    "strings"
    "testing"
)

var (
    server   *httptest.Server
    reader   io.Reader //Ignore this for now
    usersUrl string
)

func init() {
    server = httptest.NewServer(api.Handlers()) //Creating new server with the user handlers

    usersUrl = fmt.Sprintf("%s/users", server.URL) //Grab the address for the API endpoint
}

Try running the test.

go test src/api/api_test.go

There should be no error.

ok  	command-line-arguments	0.019s

Now we are going to test the success case for creating user. To do this, we are going to:

  • Create an HTTP request with JSON body

  • POST it to the endpoint we create in the init function

  • and check to make sure that the HTTP response status code is 201

If this expectation is not met, we want to error. And, here is the code.

func TestCreateUser(t *testing.T) {
    userJson := `{"username": "dennis", "balance": 200}`

    reader = strings.NewReader(userJson) //Convert string to reader

    request, err := http.NewRequest("POST", usersUrl, reader) //Create request with JSON body

    res, err := http.DefaultClient.Do(request)

    if err != nil {
        t.Error(err) //Something is wrong while sending request
    }

    if res.StatusCode != 201 {
        t.Errorf("Success expected: %d", res.StatusCode) //Uh-oh this means our test failed
    }
}

That’s it. Everything is manual. You might find Go testing framework kind of lacking especially if you come from Ruby/Rails background where stubbing/mocking is the norm. I actually kind of enjoy it because everything is explicit and there isn’t that much magic going on.

I was going to write down the other two tests but I guess I will leave it as an exercise. As always, you can go to the source code and see how I did it.

Thanks for reading!

I recently picked up a new hobby: cooking. And, I am going to share with you my grilled cheese recipe.

Ingredients

  • Two slices of white (sourdough or Tuscan pane is okay too) bread. Buy a and good-quality one like Japanese Shokupan).

  • Olive oil

  • Salted butter

  • Sharp cheddar cheese (or, better yet, truffle Gouda cheese)

  • Salt

  • Pepper

Equipments

  • Skillet

  • Spatula

Instructions

Preheat your skillet on medium heat (you can start with medium-hot and lower it shortly after). Wait for around 2 minutes for your skillet to heat up and put a LITTLE bit of olive oil on it.

While waiting for your oil to heat up, prepare your bread and butter. Start by spreading a generous amount of butter on 1 side of each slice of bread. Season the buttery side of the bread with a bit of salt and a generous amount of pepper.

img

Once your skillet is hot, put the buttery side of the bread down on the skillet.

img

Grab your olive oil and spray some of it onto the non-buttery side of the bread. Season this other side with salt and pepper as well.

img

Once the buttery side has turned slightly brown, flip the bread slices over and grill the other side (until it’s slightly brown as well). Note that you do not want to fully grill both sides, because we are going to flip it again 1 more time before adding the cheese.

img

Flip both slices again, and, now, add two slices of cheddar cheese on top of one bread slice and sandwich it with the other bread slice. Make sure that the buttery side is the outer side.

img

Wait a couple minutes for the cheese to fully melt. Flip over the sandwich a couple more times to make sure that both sides are fully grilled (and crunchy).

img

Enjoy!

What is an anonymous field?

An anonymous field is a field in a struct definition that has a type but does not have a name. Below is an example of declaring an anonymous field, notice how Human struct has Animal defined but without a name.

type Animal struct {
    Name string
}

type Human struct {
    *Animal
}

Inheritance and overriding

One use case of anonymous field is to replicate the behavior of method/field inheritance and overriding in object-oriented languages. In the example below, I define a Speak() method for Animal but not for Human. However, because Human has a *Animal as an anonymous field, calling Speak() on a Human will call the Animal’s Speak method. In this case, Speak is called promoted method

//Continuing with examples above

func (a * Animal) Speak(){
    fmt.Println("Animal speaks")
}

func main() {
    a := &Animal{
        Name: "Dennis",
    }

    b := &Human{a}

    a.Speak() //Animal speaks
    b.Speak() //Animal speaks
}

We can also override Animal’s Speak method by defining a method with the same method signature for Human. Note there is no encapsulation in that we can still access Animal from Human and call its Speak method.

func (a * Animal) Speak(){
    fmt.Println("Animal speaks")
}

func (h * Human) Speak(){
    fmt.Println("Human speaks")
}

func main() {
    a := &Animal{
        Name: "Dennis",
    }
    b := &Human{a}
    a.Speak() //Animal speaks
    b.Speak() //Human speaks

    //Note here that we are still able to access Animal's Speak()
    b.Animal.Speak() //Animal speaks
}

The same logic also applies when you try accessing a field that is not explicitly defined by the containing struct (in this case Human). That is, if you try accessing Name on a Human, you are going to get Animal’s Name instead.

fmt.Println(b.Name) //Dennis

Name conflict

A struct definition can contain more than two anonymous fields and this can result in an error if not used carefully. Imagine what would happen if both anonymous fields have a field with the same name. Take a look at the example below.

type Animal struct {
    Name string
}

type Pet struct{
    Name string
}

type Human struct {
    *Animal
    *Pet
}

func main() {
    b := &Human{&Animal{Name: "Dennis"}, &Pet{Name: "Zedo"}}
    fmt.Println(b.Name); //Error
}

The above code will give you an error because calling b.Name is ambiguous and Go does not know which Name the code is referring to. You can fix the above code by being more explicit and specify the anonymous field before the selector.

//...
fmt.Println(b.Animal.Name);
fmt.Println(b.Pet.Name);
//...

Use case of anonymous methods

When writing API endpoints, you often have to parse values from the request passed in to the handler function. One easy way to do this is to create functions that would take a pointer to the request as a parameter and return the appropriate values (IP address, API version in header, etc).

func IpAddress(r * http.Request) string {
    return strings.Split(r.RemoteAddr, ":")[0]
}

func ApiVersion(r * http.Request) uint {
    //.....
}

func ValidRequest(r * http.Request) bool {
    //.....
}

A better way to do this, in my opinion, is to use anonymous field to extend the functionality of http.Request so that you can define new methods on the containing struct and still have access to http.Request methods and fields.

type APIRequest struct {
    *http.Request
}

Defining new methods for APIRequest

func (a * APIRequest) IpAddress() string{
    return strings.Split(a.RemoteAddr, ":")[0]
}

func (a * APIRequest) ApiVersion() uint{
    //logic that acts on `a`
}

func (a * APIRequest) Valid() bool{
    //logic that acts on `a`
}

And, this is how I would use APIRequest

func SessionHandler(writer http.ResponseWriter, request *http.Request ) {
    apiReq := &APIRequest{request} //Wrapping *http.Request inside APIRequest

    fmt.Println(apiReq.IpAddress()) //Using extended method

    if !apiReq.Valid() {
        //Error
    }

    fmt.Println(apiReq.Header) //Still able to access *http.Request field
}

Given all the examples above, it’s tempting to think of anonymous field as inheritance’s identical twin. However, with its minimal syntactic sugar, anonymous field provides a lot more flexibility when trying to extend behavior of a struct. One thing that I find very helpful is that, rather than thinking of anonymous field a parent-child relationship, trying thinking of it as a wrapped or decorated object.

Continuing with the blog example from part 1 and 2, I am going to add a feature that lets visitors to post a message and save it to the DB.

Creating Message model and controller

Before creating the model, I am going to set up the Rails application to use MySQL locally.

Add the mysql2 gem to Gemfile and run bundle install:

#Gemfile
#...
gem mysql2
#...

Modify database.yml content to the following:

#database.yml
development:
  adapter: mysql2
  database: blogapp_development
  username: root
  pool: 5
  timeout: 5000

Now, create the message model along with its controller by running the following under the blogapp folder.

rails generate model message name:string content:string
rails generate controller messages
rake db:create
rake db:migrate

At this point, You should be able to go to MySQL and see the blogapp_development database and the messages table.

Message form

In order for us to test our new feature, we need to create a form that allows users to submit messages and a page where users can view others’ messages. Update the following files:

views/messages/new.html.erb

<%= form_for @message do |f| %>
    Name: <%= f.text_field :name %><br/>
    Content: <%= f.text_field :content %><br/>
    <%= f.submit %>
<% end %>

views/messages/index.html.erb

<% @messages.each do |message| %>
    Author: <%= message.name %> <br/>
    <%= message.content %><br/><br/>
<% end %>

<%= link_to "New Message", new_message_path %>

app/messages_controller.rb

class MessagesController < ApplicationController
  def index
    @messages = Message.all
  end

  def new
    @message = Message.new
  end

  def create
    Message.create(message_params)
    redirect_to messages_path
  end

  private
  def message_params
    params.require(:message).permit(:name, :content)
  end
end

config/routes.rb

Blogapp::Application.routes.draw do
  root "messages#index"
  resources :messages
end

Start your Rails app locally, go to localhost:3000/messages/new and make sure that you can post a message and view it on localhost:3000/messages.

Creating MySQL RDS instance in AWS

Log in to your AWS account and click on the RDS link. In there, you are going to find a big blue button labeled Launch DB Instance. Click it.

img

Step 1. You are going to choose the database engine. Select MySQL Community Edition.

Step 2. Is this DB going to be used for production? Well, yes, but since this is only for demonstration purposes, we are going to select No so that we can use the free usage tier.

img

Step 3. You are going to configure the username, password, instance size etc. Take note of your username and password because you are going to need it later. Feel free to allocate more storage for the instance, but know that it will cost you more.

img

Step 4. More configuration you have to set, but really the important one for us right now is just going to be the DB name. In this step, you will also see configuration for the DB Security Group(s). Leave this to default for now. We are going to revisit this later.

img

Step 5. This is where you can configure automatic backups. Feel free to disable this but I am just going to leave it as-is

Step 6. Review and launch the instance!

img

In step 4, we left the DB instance to use the default security group. We want to set this security group so that we can use the mysql command to access the DB console.

If you are not already on the RDS Dashboard, go there and click on the Security Groups link on the side navigation bar.

You will see the list for RDS security groups here, but you will most likely only see one which is the default. Click on the magnifying glass right next to default to show its details.

img

Add a new rule to allow access to your DB from anywhere. Enter 0.0.0.0/0 for CIDR, click Add and Refresh Security Groups. It will take a 1-2 minutes for the security group’s Status to change to Authorized.

img

Once you’re done with all of the above, you can access your instance using the mysql command line. Like so:

mysql -h yourins.mydnsnameexample.rds.amazonaws.com -P 3306 -u admin -p
#You will be prompted for your password

Configure Rails to use the new RDS instance

We are done for now with the RDS Dashboard.. Now, go back to the blogapp Rails project. Open config/database.yml, and modify it appropriately with your DB instance’s name, username, password and host.

development:
  adapter: mysql2
  database: blogapp_development
  username: root
  pool: 5
  timeout: 5000

production:
  adapter: mysql2
  database: blogapp
  username: admin
  host: yourins.mydnsnameexample.rds.amazonaws.com
  password: blogappadmin
  port: 3306
  pool: 5
  timeout: 5000

Commit all of your changes and push it.

git add .
git commit -m "Form for message submission and production RDS setting"
git push origin master

Let’s go over what we just did. We created a new RDS instance and allow it to be accessible by the outside world. We also set up our Rails project to use this RDS instance in production environment. Next, we are going to update our app in EC2 to have the latest code and run migration. If everything goes smoothly, we should then be able to start our Rails app in the cloud and post messages just like we did locally.

Pull code, migrate and run in production

Log in to your EC2 instance. If your forgot how, here is the SSH command. Change the location of the .pem file and the public DNS to yours.

ssh -2 -i ~/location/to/your/blogapp.pem ubuntu@ec2-12-345-67-891.compute-1.amazonaws.com

Go to the blogapp app folder and git pull origin master to update the code.

img

Run the following to update gems and run migration

bundle install
rbenv rehash
rake db:migrate RAILS_ENV=production

You should see that the messages table that we created earlier is now created in the MySQL RDS instance. The last step is to restart the Passenger server

rbenv sudo passenger stop  -p80
rbenv sudo passenger start -p80 -d -e production

Open up a browser and enter the EC2 instance address (Public DNS). You should see the updated page with the New Message link. Try creating a new message and see it show up on the messages index page.

That’s it! You now know how to manually set up a server to host your Rails application. I hope that was not too bad. Again, send me an e-mail if you have any questions.

More: assets pre-compilation

I skipped this step for the example above because we really do not have any meaningful assets in our app. Our app does not use any style or javascripts. In real world, of course, this is not going to be the case. If you have any JS or CSS in your repo that you use, you will need to run the following before you run your Passenger server.

rbenv sudo rake assets:precompile RAILS_ENV=production

In this part, you are going to get your EC2 instance set up and ready to host your Rails application.

Creating EC2 instance

Log in to your AWS account and go to the EC2 management console by clicking the link shown below:

img

Click on the blue Launch Instance button to get to the wizard window where you can configure a new EC2 instance. The first step is selecting the AMI and we are going to use Ubuntu Server 12 LTS. Choose 64-bit and click Select.

img

Step 2, you are going to be prompted to choose the instance type which will also determine the size of CPU/Memory for your instance. Choose whichever you like, but MAKE SURE THAT YOU UNDERSTAND THE COST IMPLICATION OF SELECTING BIGGER INSTANCES. Or, for demonstration purposes, choose the cheapest and smallest type which is t1.micro. Then, click Next: Configure Instance Details button

img

I am going to skip step 3 for this tutorial and just use the default values. So, goa ahead and click Next: Add Storage.

In step 4, you can configure the volume size of your instance(s). The number will vary depending on the application’s need, but 50GB should be more than enough for the blog app that you created in Part 1. Once you are done, click Next: Tag Instance

img

In step 5, you can set key-value pairs metadata for yor application. This is useful if you want to categorize your instances by environment or by other variables. Name your instance blogapp and add a new tag named Environment and set its value to production. Once you are done, click Next: Configure Security Group

img

Step 6, this is where you can create rules for incoming traffic (by configuring which IPs and/or ports are going to be accessible). Because your instance is going to be hosting a web application and running a web server, you will want to open port 80. And, because you want to be able to SSH to it, you also want to open up port 22. SSH is added by default, so click Add Rule and choose HTTP for the protocol to open port 80. The wizard will automatically save this security group setting for you. Give it a different name and call it webserver. Make sure your setting looks like the following:

img

Click Review and Launch and you will be taken to a page to review your instance settings. Go ahead and click Launch. A pop up will appear asking you to create or select an existing key-pair which wil allow you to SSH to your instance. I am going to create a new key-pair and name it blogapp. Download the key-pay and click Launch Instance

img

Go to View Instances, and make sure that you see the newly created instance blogapp.

img

SSHing to your instance

After you have created your instance, you want to SSH to it and install packages that are essential to running your web app. The first thing that you need is the address of your instance. To get this, you need to go to the EC2 instance management console, select the blogapp instance, click the Description tab, and locate the Public DNS

img

The Public DNS should look something like ec2-12-345-67-891.compute-1.amazonaws.com. If you try to ssh to it (go ahead and give it a try), you are going to get a Permission Denied error. You need to use the private key that you downloaded earlier at the end of the instance configuration to log in to your instance.

ssh -2 -i ~/Downloads/blogapp.pem  ubuntu@ec2-12-345-67-891.compute-1.amazonaws.com

Make sure that you change the location of the pem file and the hostname appropriately. Note that you are logging in as ubuntu user. This is the default user that gets created when you start a new instance with Ubuntu AMI.

Warning: Unprotected Private Key File

The above ssh command would often fail because the permission of your private key file is too open. You need to make it more restrictive by running the following command

sudo chmod 600 ~/Downloads/blogapp.pem

Installing essential packages

Once you are able to log in to your instance, you will want to install packages that will allow your pull your code and deploy the app. Run the following on commands:

sudo apt-get update
sudo apt-get install build-essential libopenssl-ruby1.9.1 libssl-dev zlib1g-dev libmysqlclient-dev libpcre3-dev libcurl4-openssl-dev mysql-client-core-5.5 openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison nodejs subversion

The second command installs quite a bit, so please remove/add as necessary according to your app’s need.

In the following steps, you are going to install Ruby with the desired version on our Ubuntu instance. I am going to use Rbenv for managing the Ruby installations, but feel free to use RVM instead.

To read more about Rbenv, go to https://github.com/sstephenson/rbenv and why choose Rbenv over RVM.

Install rbenv

git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.profile
echo 'eval "$(rbenv init -)"' >> ~/.profile

Install ruby-build

cd ~
git clone https://github.com/sstephenson/ruby-build.git
cd ruby-build
sudo ./install.sh

Install rbenv-sudo

mkdir ~/.rbenv/plugins
git clone git://github.com/dcarley/rbenv-sudo.git ~/.rbenv/plugins/rbenv-sudo

Restart Shell

cd ~
exec $SHELL -l

Install Ruby and Bundler

rbenv install 2.0.0-p353
rbenv global 2.0.0-p353
gem install bundler

Generating SSH key and add it to Bitbucket/Github

If you created your app repository as private on Bitbucket/Github, you would not be able to clone it inside your Ubuntu instance unless you add the SSH key of your instance to your Github/Bitbucket profile. If your repository is public, you can skip this step entirely.

Create an SSH key by running the following command

ssh-keygen

Keep pressing enter until it has generated your ssh keys. The key generation should look something like this:

img

Your new SSH key should be in ~/.ssh/id_rsa.pub. Copy the content of that file and add it to your list of SSH keys in Github/Bitbucket.

Cloning your app and starting the server

Once you have added your SSH key, clone your repository to your home directory (Note: real-world scenarios often use /var/apps to store its application codes) and install all of the necessary gems

cd ~
git clone git@bitbucket.org:denniss/blogapp.git
cd ./blogapp
rbenv rehash
bundle install
rbenv rehash

To learn more about rbenv rehash

Once you have installed all the gem dependencies, you are ready to start your app

rbenv sudo passenger start -p80 &

The previous command is going to install all Passenger’s dependencies on your instance (which will take a while) and start the Passenger server. The output should more or less look like this:

img

Once Passenger is and started, you can see your app by entering the Public DNS to your browser. You should see the Hello World that you created from Part 1

img

There you go. Now you have your Rails app running in the cloud :)

Continue to part 3

If you have any questions/comments, please send me an e-mail at dennis.suratna@gmail.com