Jul 20, 2020 Tutorials

Servers Deep Dive Part 2: Launching a Server

Take your scene building to the next level by learning how you can build a server that a Decentraland scene can interact with.

Nico Earnshaw

Share

In part 1 of this series we saw how our scene fetches data from an existing API. That’s going to be very valuable knowledge when we take things to the next level.

Question: What if we mount our own API, and use that to store information that affects how the scene is displayed for other players who come in later?

There are a number of services out there that you can use – many of them for free up to certain volumes, which in many cases might well be enough for using in a scene. Ultimately, if you want to handle a really high volume of users, you will likely have to pay a fee eventually. Some services offer storage, others processing power, some offer both together.

A good option to get started is Google Firebase, since it has a friendly interface and provides both processing power and storage for free up to certain usage volumes. It also provides https encryption out of the box, which is one less thing to worry about.

There are many other options out there, and just as many strong opinions about which are the best. Much has been written about it, and a lot of that applies just as much to Decentraland scenes as to typical web applications. Some other alternatives that you might want to try out are Heroku, Azure and Digital Ocean. Vercel Now (formerly known as Zeit Now) is free and a good option for running ‘serverless’ lambdas. It also comes with https support but doesn’t provide any data persistence, so for most scenarios you’ll want to pair it up with a storage service like Amazon S3, Azure Storage, Google Cloud Storage, DigitalOcean Spaces, etc.

Through the majority of this post, we’ll be guiding you through an example using Firebase only. This is going to get a lot more hands on, so I recommend you roll up your sleeves and follow these steps to learn by doing!

Guestbook scene

We’ll be building an API to use on this simple example scene where players can read and sign a guestbook. When a player opens the guestbook, the scene needs to send out a request to our API, that should return the full list of players that have signed. When a player signs the book, the scene should send another request to our API so that their name is added to the book.

The scene is a tribute to all of those tourist trap locations that spoil the very thing they’re meant to be all about. The guestbook itself could of course be used in a non-ironic way anywhere else, feel free to copy it onto your own scenes if you want to have that functionality.

https://github.com/decentraland-scenes/Guest-Book-API

Before we begin

Let’s embark on a journey to create a Firebase server. But before you get started, make sure you have the following:

• Postman installed
• Have a Google account set up

Create a Firebase project

If you don’t have a Firebase account already, start by creating one. Go to the Firebase console and click Add Project.

Give your project a name, and select a location to host it.

Build the project scaffolding

Install the Firebase CLI, by running:

npm install -g firebase-tools

Open your command line tool of choice and navigate to the folder where you want to create your server project. Once there, run firebase init.

Here you’ll be presented with various different services that are offered by Firebase. For this example all we really need is the Functions option. So hit spacebar to select Functions and then enter to continue.

The CLI then presents some more questions:

• Use an existing project and select the one you just created
• Select TypeScript as a language, as this is what we’ll be using in this tutorial and in most example scenes
• Linting is up to you. We recommended it unless you have something already configured that might be in conflict
• Install the dependencies automatically

Great! At this point we should have the project all set up and ready to start working. Take a look at the new files and folders that were created by this process. You should now see a functions folder with the following files in it:

• src/index.ts
• package.json
• tsconfig.json
• node_modules folder
• lib folder

The only one of these files that we’ll be editing in this tutorial is the index.ts file. You can mostly ignore the rest – just know they’re there.

Creating a first endpoint

Inside our index.ts file, we’ll be using the popular Express library to mount a ‘serverless’ API.

About ‘serverless’ APIs: What’s commonly referred to as a ‘serverless’ API can elicit some confusion, because you really are setting up a server, after all. What’s different between a ‘serverless’ API and a traditional server hosting, is that the environments where your code runs are completely managed by the provider, you just need to write your code, and not worry about things like updating the OS of the machine that runs it. It also means that you don’t have a machine or a portion of a machine that’s continually dedicated to running your code. Instead, a virtual machine is instanced on demand whenever your API endpoints are hit, and then when the work is done those resources are returned back to the sea of brain that they came from. This generally means cheaper hosting, since the service providers can pool resources with much more efficiency.

We recommend changing the Node version used on the project to version 10. By default your project uses Node version 8, which works fine today, but will eventually be deprecated by Firebase. To change that, open package.json and edit the line that says ”node”: “8” so that it says ”node”: “10”.

"engines": {
   "node": "10"
 }

Open the index.ts file and overwrite the default code with the following lines to create your first API endpoint:

const functions = require('firebase-functions')
const express = require('express')
const cors = require('cors')

const app = express()
app.use(cors({ origin: true }))

app.get('/hello-world', (req: any, res: any) => {
  return res.status(200).send('Hello World!')
})

exports.app = functions.https.onRequest(app)

This script relies on a couple of dependencies that must be installed, so on the terminal do cd to the functions folder, and then run the following commands:

npm i express
npm i cors

The first thing this code does is import three libraries:

firebase-functions: lets you create serverless functions to run on Firebase express: lets you create a server instance to handle the requests cors: Makes the content returned by your endpoints accessible to external sites (like Decentraland, for example)

After that, we’re instancing an Express router to handle the routes of our endpoints, and we’re setting CORS policies so that the content returned by these endpoints is open to everyone.

Then comes the more interesting part. app.get('/hello-world', (req: any, res: any). makes the Express router listen for GET HTTP requests to the /hello-world sub-url. It will run the function that follows every time it receives one of those requests. We’re also defining a req object, that contains all of the additional data in the request, and a res object, that will be sent as a response to the caller.

Every time this endpoint is called, we return a ‘Hello World!’ string, with an HTTP status code of 200, which is universally recognized as a success response.

Finally, exports.app = functions.https.onRequest(app) exposes the express application to Firebase functions.

We’re finally ready to try our app! We can run it locally by executing the following in the command line from the functions folder:

npm run serve

Note: If you have any issues with this step, in the /functions folder you’ll see two new files firebase-debug.log and ui-debug.log which include more verbose information than the log you see on console, these might be quite useful.

Okay, so our API is running, but where? What’s the full address we need to call? The terminal responds with some info, including a localhost and a port of an emulator. If you’re thinking that’s it, you’re on the right path but not all the way there. Firebase makes the URL a bit of a mouthful to be honest. Here’s how you put it together:

[<------domain--->]/[<-app id->]/[<-zone->]/app/[<-endpoint->]
http://localhost:5001/dcl-guestbook/us-central1/app/hello-world

You might be wondering what your app’s ID is, and you might not remember what zone you selected. But if you look closer at what the terminal responds, you’ll find a URL for an Emulator UI: http://localhost:4000/functions.

Paste that into a browser tab and you’ll find the full path to your API.

You can also look up your app ID and deployment zone in your Project settings page:

So now we can use Postman to send a GET request to our newly exposed API, at http://localhost:5001/dcl-door/us-east1/app/hello-world (or, since this is a simple GET request, you could also just use a browser). Remember that the address may differ depending on your project name or region.

Upon sending the request, you should see that our server responds to Postman with Hello World!.

If you made it this far, congratulations – you already have your server running!

Set up the database

The reason why we are creating a server is to be able to store and retrieve data that other players uploaded. For that we’ll need a database.

Firebase offers two different options: a traditional database called Realtime database, and a NoSQL database called Cloud Firestore. For this tutorial we’ll go with Cloud Firestore.

About NoSQL Databases: In a NoSQL database, data is stored in documents inside collections, and each document can have a different structure from others, so they don’t necessarily fit into the columns and rows of a table like a traditional SQL database and because of that they tend to be more lightweight and to perform better.

Go to your project page on Firebase and open the Database tab. Click Create database to create your initial database.

Select ‘Start in test mode’ to enable all reads and writes easily. Also, select a region for hosting the database, which can be the same region where you host the app or not.

Remember that our functions server runs as a serverless ephemeral thing that only exists while someone hits its endpoints – and that will be a separate machine from the Firestore database which will hold persistent data. Any requests that need to go from the functions server machine to the db server machine need to be authenticated somehow.

We could configure our database so that different users with Google accounts can edit the data in it, but that’s not going to be of any use to us since players aren’t going to do a login to their Google account from inside Decentraland. What we need instead is to create a Service account. A Service account is a fake user that is used for app-to-app communication. Our functions server is going to impersonate that user when it connects to the Firestore database.

We’re going to create a Service account, retrieve its keys and store those in our project.

Go to your app in the Firebase Console, click the little gearbox and select Users and permissions. Then open the Service accounts tab.

In this screen, keep the Node.js option selected and click Generate new private key. This will download a .json file to your computer with the secret credentials for this service account.

Copy this file into your project folder and rename it if you wish. Make sure you don’t share the service account credentials publicly anywhere. In this case, I added this file to the .gitignore file, so that it doesn’t get uploaded to the GitHub repo.

Then, back on the Service Accounts page, you’ll note that there’s a really helpful snippet of code written out wich already points to your database’s URL. You can copy those lines of code straight into the end of your index.ts file, and that’s all you need to make this work! Just make sure that the path to the .json file with the keys matches where you’re really storing the file.

var admin = require('firebase-admin')

var serviceAccount = require('path/to/serviceAccountKey.json')

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount),
  databaseURL: 'https://dcl-door.firebaseio.com',
})

These lines of code import the Firebase module that takes care of permissions, then it loads your credentials file and initializes an authenticated application that we can later use in our code.

We’ll add one more line of code after that, which initiates an authenticated db object to interface with Firestore.

const db = admin.firestore()

Now let’s return to the Database tab of our project dashboard on Firebase. We’ll create our first document of data manually. Remember, in a NoSQL database, data is stored into collections, and each contains a list of documents. In our case, each signature will be stored as its own document, containing a name and an id field. These documents will all belong to a single collection.

Click Start collection, and name your collection Signatures, then create a first signature in that collection. Click AutoID to generate a random string for a doc ID, and give it two fields of type string: id and name.

Make database calls

With this we are ready to start building the real functionality that we had set out to build. There’s essentially two things we need to do:

1. Retrieve the list of signatures when someone reads the book
2. Store a new signature when someone signs the book

Add the following code to index.ts, to handle those two actions:

let signatures = db.collection('Signatures')

app.get('/get-signatures', async (req: any, res: any) => {
  try {
    let response: any = []
    await signatures.get().then((queryResult: { docs: any }) => {
      for (let doc of queryResult.docs) {
        response.push(doc.data())
      }
    })
    return res.status(200).send(response)
  } catch (error) {
    console.log(error)
    return res.status(500).send(error)
  }
})

app.post('/add-signature', async (req: any, res: any) => {
  let newSignature = req.body
  try {
    await signatures
      .doc('/' + Math.floor(Math.random() * 100000) + '/')
      .create({
        id: newSignature.id,
        name: newSignature.name,
      })
    return res.status(200).send('Signed book!')
  } catch (error) {
    console.log(error)
    return res.status(500).send(error)
  }
})

TIP: If you’re not familiar with some of the syntax here, we explain what try, catch, async and await mean in the previous post of this series. You can also find info about Firebase’s API in their documentation.

The first line here retrieves an object that represents the Signatures collection that we created through the UI. We’ll use this object to read and write to this collection. We then create two new endpoints, one listens for GET requests on /get-signatures, the other listens for POST requests on /add-signature, and we define functions to run on each of the two events.

The first of the two endpoints runs a query on the db to get all of the docs in the collection, then packs all of these into an array that it sends back in the response message.

Let’s start up our server again with npm run serve and give it a try…

The response we get is a list of a single item, the one we added earlier through the UI.

The second endpoint expects the incoming request to have a body containing the user ID and name. This information comes into our function through the req object, so we can easily pick it up doing let newSignature = req.body. We then create a new document for the signatures collection, giving it a random number as an ID and populating it with the data from the request body.

When calling this endpoint through Postman, we need to remember to set POST as the method, and write data into the Body of the request.

If it all goes well, we get a response that says “Signed book!” If we now do another request to the /get-signatures endpoint, we should see both our entries.

One cool feature from Firebase is that if we go to our project’s Database tab, you can see new data being updated in real time in the UI. So if we open that screen, we should see both documents listed there.

Deploy the server

All of our functions work perfectly when running them locally through npm run serve. Now there’s one last step remaining: deploying our server to the cloud! All you have to do is run this on the console:

firebase deploy

The URLs to reach your endpoints are now different. Once again, Firebase makes the URL structure a bit of a mouthful. This is how you build one:

[<---zone + app id + cloudfunctions.net----->]/app/[<--endpoint-->]
https://us-central1-dcl-guestbook.cloudfunctions.net/app/hello-world

You can also check what the full URL looks like if you open the Functions tab in your project’s Firebase dashboard.

Test out the new path of your endpoints on Postman to make sure everything works as expected.

You can now start building your scene around this API. We won’t cover that in this tutorial, but it shouldn’t be too different from what we covered in Part 1 of this series.

You can also look at what we’ve done in the project repo: github.com/decentraland-scenes/Guest-Book-API

Leaderboard example scene

Check out this other example scene. It’s quite similar but slightly more elaborate and might be exactly what you need for your scene. Feel free to borrow any bits of code you want from it!

github.com/decentraland-scenes/Leader-Board

Combine Firebase with another storage service

For Genesis Plaza, we used something similar when it came to displaying messages that were written by players on the tower. We used Firebase to expose our endpoints but we actually did the storing of data on an Amazon S3 server rather than on Firebase’s Firestore service. Requests to change the data go through our Firebase server, but then when players fetch data to read, they get that straight from the Amazon S3 server.

Why do this? Firebase is a pretty good solution, but an Amazon S3 server is more robust and also remains a cheaper solution when dealing with many requests. Although we can read the data of a .json file in the S3 server directly from a Decentraland scene, we can’t write to it directly from the scene. This is because you need Amazon credentials to do the writing and we don’t recommend keeping those in a scene’s code as they’re not entirely safe there, so we handle that part from our Firebase server.

Check out the repo for the Plaza: github.com/decentraland-scenes/Genesis-Plaza

Combine a server with the Message Bus

Instead of having players polling for new values in the server every couple of seconds, you could instead combine using the server and the scene Message Bus. We’ve covered how we use the MessageBus on this blogpost about Genesis Plaza. With this approach, you can reduce the number of requests that get sent to your server and you can keep players in much better sync with what each other is doing.

github.com/decentraland-scenes/Zenquencer

In this example you can play with a sequencer to write out some musical patterns and then play them back. When a player comes into the scene they download the latest pattern from off the server. Then, as different players that are there change the pattern, they get these changes from each other using the Message Bus, they don’t need to check the server regularly to know what’s new.

For this to work properly, we need to keep a separate version of this pattern for each realm and know what realm each player is on when they update the pattern. This is because only players that are in the same realm message each other via the Message Bus. There would otherwise be odd inconsistencies in what the pattern ends up being when players that are in different realms modify the same pattern without notifying each other. The scene includes the player’s realm as part of the requests it sends, and the server then handles a different .json file depending on the realm.

Another noteworthy thing we’re doing in this example is that changes aren’t sent to the server right away, but instead we do a little buffer using the utils.Delay component, so that if the player changes several notes in quick succession, the server only gets notified of the final state of the pattern. This helps reduce the number of requests that the server needs to handle. For it to work, each update request needs to send the full state of the pattern, rather than just the changed elements.

This other example includes a mural where pixels can be painted different colors. It uses the Message Bus and server very much in the same way as the sequencer we just shared.

github.com/decentraland-scenes/mural-example-scene

Here’s an even cooler example that lets you build with placing 3D voxels. From the server’s point of view, it’s really quite similar to the mural, except that it deals with 3D coordinates rather than 2D coordinates.

github.com/decentraland-scenes/voxel-art-creator

A note about security: None of the examples we have shared implement any security measures against malicious use of the APIs. Players are capable of sending fake requests to these APIs if they want to. There isn’t a lot to worry about in these examples, since it would be pretty harmless if someone did that. But if, for example, we were automatically sending an NFT reward to players who reached a certain score on a scoreboard API, then that could be abused.

You could easily add a security key, so that the scene uses that key when calling the API, but that wouldn’t add a lot of security, since players can obtain that key from the scene’s code. They could also pick it from viewing outgoing HTTP requests in the Network tab in the browser console.

Ideally, you should run validations on the server side. You could, for example, keep track of various events that happen in your game and only send a reward to players that have gone through all of those before requesting for the reward. That’s still hackable, but if they can’t see the code of the server, they won’t know what validations they need to pass and it gets to the point where it’s easier to play legitimately than to look for ways to cheat.

This was a long post, but we hope that you find yourselves equipped with new tools that enable you to create new exciting experiences in Decentraland!

Stay tuned for part 3 of this series, where we’ll discuss using Websockets servers for creating multiplayer experiences with a much more fluid and real-time feel.