Whispering Gophers

Network programming in Go

Andrew Gerrand

Francesc Campoy

Introduction

This code lab demonstrates network programming with Go.

The final goal is to build a "whispernet": a peer to peer mesh network for transmitting text messages.

Design overview

Pre-requisites

You must have these tools installed and working:

This is not an introduction to the Go Programming Language;
some experience writing Go code is required. To learn Go:

Create a workspace (see golang.org/doc/code.html) and install the support libraries:

$ go get code.google.com/p/whispering-gophers/hello
$ $GOPATH/bin/hello
It works!

Creating a workspace

To write Go code you need a workspace. Create one now if you have not already.

$ mkdir $HOME/gocode
$ export GOPATH=$HOME/gocode   # or add this to ~/.profile

Build and install a hello example program:

$ go get code.google.com/p/whispering-gophers/hello

This installs the hello binary to $GOPATH/bin.

Check that this worked:

$ $GOPATH/bin/hello
It works!

The exercises

This code lab is divided into 8 exercises. Each exercise builds on the previous one.

The skeleton directory in the whispering-gophers repository contains unfinished programs that you should complete yourself.

There is also a solution directory that contains the finished programs—only peek if you really need to! :-)

Vital resources

Visit

for code samples and support libraries.

These slides are available at:

Part 1: reading and writing

Write a program that

This line of input:

Hello, world

should produce this output:

{"Body":"Hello, world"}

This is our system's basic message format.

Readers and Writers

The io package provides fundamental I/O interfaces that are used throughout most Go code.

The most ubiquitous are the Reader and Writer types, which describe streams of data.

package io

type Writer interface {
    Write(p []byte) (n int, err error)
}

type Reader interface {
    Read(p []byte) (n int, err error)
}

Reader and Writer implementations include files, sockets, (de)compressors, image and JSON codecs, and many more.

Chaining Readers

package main

import (
    "compress/gzip"
    "encoding/base64"
    "io"
    "os"
    "strings"
)

func main() {
    var r io.Reader
    r = strings.NewReader(data)
    r = base64.NewDecoder(base64.StdEncoding, r)
    r, _ = gzip.NewReader(r)
    io.Copy(os.Stdout, r)
}

const data = `
H4sIAAAJbogA/1SOO5KDQAxE8zlFZ5tQXGCjjfYIjoURoPKgcY0E57f4VZlQXf2e+r8yOYbMZJhoZWRxz3wkCVjeReETS0VHz5fBCzpxxg/PbfrT/gacCjbjeiRNOChaVkA9RAdR8eVEw4vxa0Dcs3Fe2ZqowpeqG79L995l3VaMBUV/02OS+B6kMWikwG51c8n5GnEPr11F2/QJAAD//z9IppsHAQAA
`

Buffered I/O

The bufio package implements buffered I/O. Its bufio.Scanner type wraps an io.Reader and provides a means to consume it by line (or using a specified "split function").

package main

import (
	"bufio"
	"fmt"
	"log"
	"strings"
)

const input = `A haiku is more
Than just a collection of
Well-formed syllables
`

func main() {
    s := bufio.NewScanner(strings.NewReader(input))
    for s.Scan() {
        fmt.Println(s.Text())
    }
    if err := s.Err(); err != nil {
        log.Fatal(err)
    }
}

Encoding JSON objects

The encoding/json package converts JSON-encoded data to and from native Go data structures.

package main

import (
	"encoding/json"
	"log"
	"os"
)

type Site struct {
    Title string
    URL   string
}

var sites = []Site{
    {"The Go Programming Language", "http://golang.org"},
    {"Google", "http://google.com"},
}

func main() {
    enc := json.NewEncoder(os.Stdout)
    for _, s := range sites {
        err := enc.Encode(s)
        if err != nil {
            log.Fatal(err)
        }
    }
}

The Message type

Messages are sent as JSON objects like this:

{"Body":"This is a message!"}

Which corresponds to a Go data structure like this:

type Message struct {
    Body string
}

Error checking

Many functions in Go return an error value.
These values are your friends; they will tell you where you went wrong.
Ignore them at your peril!

Use log.Println to print log messages, and log.Fatal to print a message and exit the program printing a stack trace.

package main

import (
	"compress/gzip"
	"log"
	"strings"
)

func main() {
    log.Println("Opening gzip stream...")
    _, err := gzip.NewReader(strings.NewReader("not a gzip stream!"))
    if err != nil {
        log.Fatal(err)
    }
    log.Println("OK!")
}

Part 1: reading and writing (recap)

Write a program that

This line of input:

Hello, world

should produce this output:

{"Body":"Hello, world"}

This is our system's basic message format.

Part 2: Send messages to a peer

Extend your program:

Flag

The flag package provides a simple API for parsing command-line flags.

package main

import (
    "flag"
    "fmt"
    "time"
)

var (
    message = flag.String("message", "Hello!", "what to say")
    delay   = flag.Duration("delay", 2*time.Second, "how long to wait")
)

func main() {
    flag.Parse()
    fmt.Println(*message)
    time.Sleep(*delay)
}
$ flag -message 'Hold on...' -delay 5m

Making a network connection

The net package provides talk/code for network operations.

The net.Dial function opens a nework connection and returns a net.Conn, which implements io.Reader, io.Writer, and io.Closer (or io.ReadWriteCloser).

package main

import (
	"fmt"
	"io"
	"log"
	"net"
	"os"
)

func main() {
    c, err := net.Dial("tcp", "www.google.com:80")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Fprintln(c, "GET /")
    io.Copy(os.Stdout, c)
    c.Close()
}

(Usually you would use the net/http package to make an HTTP request; the purpose of this example is to demonstrate the lower-level net package.)

Part 2: Send messages to a peer (recap)

Extend your program:

Part 3: Serving network connections

Write a new program:

Listen/Accept/Serve (1/2)

The net.Listen function binds to a socket and returns a net.Listener.
The net.Listener provides an Accept method that blocks until a client connects to the socket, and then returns a net.Conn.

This server reads data from a connection and echoes it back:

package main

import (
	"fmt"
	"io"
	"log"
	"net"
)

func main() {
    l, err := net.Listen("tcp", "localhost:4000")
    if err != nil {
        log.Fatal(err)
    }
    for {
        c, err := l.Accept()
        if err != nil {
            log.Fatal(err)
        }
        fmt.Fprintln(c, "Welcome to the echo server!")
        io.Copy(c, c)
    }
}

Goroutines

Goroutines are lightweight threads that are managed by the Go runtime. To run a function in a new goroutine, just put "go" before the function call.

package main

import (
    "fmt"
    "time"
)

func main() {
    go say("let's go!", 3*time.Second)
    go say("ho!", 2*time.Second)
    go say("hey!", 1*time.Second)
    time.Sleep(4 * time.Second)
}

func say(text string, duration time.Duration) {
    time.Sleep(duration)
    fmt.Println(text)
}

Listen/Accept/Serve (2/2)

To handle requests concurrently, serve each connection in its own goroutine:

package main

import (
	"fmt"
	"io"
	"log"
	"net"
)

func main() {
    l, err := net.Listen("tcp", "localhost:4000")
    if err != nil {
        log.Fatal(err)
    }
    for {
        c, err := l.Accept()
        if err != nil {
            log.Fatal(err)
        }
        go serve(c)
    }
}

func serve(c net.Conn) {
    fmt.Fprintln(c, "Welcome to the echo server!")
    io.Copy(c, c)
}

Decoding JSON

Decoding JSON from an io.Reader is just like writing them to an io.Writer, but in reverse.

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

type Site struct {
    Title string
    URL   string
}

const stream = `
    {"Title": "The Go Programming Language", "URL": "http://golang.org"}
    {"Title": "Google", "URL": "http://google.com"}
`

func main() {
    dec := json.NewDecoder(strings.NewReader(stream))
    for {
        var s Site
        if err := dec.Decode(&s); err != nil {
            log.Fatal(err)
        }
        fmt.Println(s.Title, s.URL)
    }
}

Part 3: Serving network connections (recap)

Write a new program:

Part 4: Listening and dialing

Part 5: distributing the listen address

Add an Addr field to Message

Add an Addr field to the Message type:

type Message struct {
    Addr string
    Body string
}

Now, when constructing Message values, populate the Addr field with the listen address:

{"Addr":"192.168.1.200:23542","Body":"This is a message!"}

Obtaining the listener address (1/2)

The net.Listener interface provides an Addr method that returns a net.Addr.

package main

import (
    "log"
    "net"
)

func main() {
    l, err := net.Listen("tcp", ":4000")
    if err != nil {
        log.Fatal(err)
    }
    log.Println("Listening on", l.Addr())
}

When listening on all interfaces, as specified by the empty hostname in the string ":4000" above, the net.Addr won't be that of our public IP address.

To complete our program, we need to find that IP.

Obtaining the listener address (2/2)

The "code.google.com/p/whispering-gophers/util" package provides a Listen function that binds to a random port on the first available public interface.

package main

import (
    "log"

    "code.google.com/p/whispering-gophers/util"
)

func main() {
    l, err := util.Listen()
    if err != nil {
        log.Fatal(err)
    }
    log.Println("Listening on", l.Addr())
}

Part 5: sending the listen address (recap)

Part 6: separate reading and writing

Channels

Goroutines communicate via channels. A channel is a typed conduit that may be synchronous (unbuffered) or asynchronous (buffered).

package main

import "fmt"

func main() {
    ch := make(chan int)
    go fibs(ch)
    for i := 0; i < 20; i++ {
        fmt.Println(<-ch)
    }
}

func fibs(ch chan int) {
    i, j := 0, 1
    for {
        ch <- j
        i, j = j, i+j
    }
}

Part 6: separate reading and writing (recap)

Part 7: tracking peer connections

Sharing state

Mutexes are a simple means to protect shared state from concurrent access.

package main

import (
	"fmt"
	"sync"
	"time"
)

func main() {
    var (
        count int
        mu    sync.Mutex // protects count
    )
    for i := 0; i < 10; i++ {
        go func() {
            for {
                mu.Lock()
                count++
                mu.Unlock()
                time.Sleep(5 * time.Millisecond)
            }
        }()
    }
    time.Sleep(time.Second)
    mu.Lock()
    fmt.Println(count)
    mu.Unlock()
}

Tracking peer connections (1/2)

Each peer connection runs in its own goroutine.

Each goroutine has its own chan Message. It reads messages from the channel, and writes them to the connection as JSON objects.

A central peer registry associates each peer address with its corresponding chan Message.

type Peers struct {
    m  map[string]chan<- Message
    mu sync.RWMutex
}

Tracking peer connections (2/2)

Before making a peer connection, ask the peer registry for a channel for this address:

// Add creates and returns a new channel for the given address.
// If an address already exists in the registry, it returns nil.
func (p *Peers) Add(addr string) <-chan Message

When a peer connection is dropped, remove the peer from the registry:

// Remove deletes the specified peer from the registry.
func (p *Peers) Remove(addr string)

To broadcast a Message to all peers, ask the peer registry for its list of chan Message and send the Message to each channel

// List returns a slice of all active peer channels.
func (p *Peers) List() []chan<- Message

Part 7: tracking peer connections (recap)

Part 8: connect to multiple peers

Sending without blocking

If a peer connection stalls or dies, we don't want to hold up the rest of our program. To avoid this problem we should do a non-blocking send to each peer when broadcasting messages. This means some messages may be dropped, but in our mesh network this is okay.

package main

import "fmt"

func main() {
    ch := make(chan int)

    select {
    case ch <- 42:
        fmt.Println("Send succeeded")
    default:
        fmt.Println("Send failed")
    }
}

Part 8: connect to multiple peers (recap)

Part 9: re-broadcast messages

Add ID to Message

Add an ID field to the Message type:

type Message struct {
    ID   string
    Addr string
    Body string
}

Now, when constructing Message values, populate the ID field with a random string:

{"ID":"a09d2abb1ad536ada",Addr":"192.168.1.200:23542,"Body":"This is a message!"}

Generating random strings

Use the util.RandomID function to generate a random string.

package main

import (
    "fmt"

    "code.google.com/p/whispering-gophers/util"
)

func main() {
    id := util.RandomID()
    fmt.Println(id)
}

Tracking message IDs

To track messages IDs, use a map[string]bool. This works as a kind of set.

seen := make(map[string]bool)

To check if an id is in the map:

if seen[id] {
    fmt.Println(id, "is in the map")
}

To put an id in the map:

seen[id] = true

You should implement a function named Seen — make sure it is thread safe!

// Seen returns true if the specified id has been seen before.
// If not, it returns false and marks the given id as "seen".
func Seen(id string) bool

Part 9: re-broadcast messages (recap)

Thank you

Andrew Gerrand

Francesc Campoy