Storing Data

Because having an intern enter secrets into a spreadsheet would mean I'd have to write a job description.

Let's talk about project structure for a second. Thruster doesn't prescribe any sort of methodology for how you should organize your code, but I'll sprinkle in how I tend to organize mine. Feel free to ignore this bit in italics. I usually choose one of two ways: 1. Domain driven, where a folder contains the routes, models, and maybe some associated services or helpers. For example, a users folder would have all of the routes for getting, creating, updating a user, as well as the user model and methods for creating a user in the database.

2. MVC (or rather, MCS perhaps?) I'll setup a folder for models, controllers (or routes,) and services. Models are simple structs and methods for handling them in the storage layer. Controllers (or routes) are the code for taking an external request and making service calls and model updates. Services is a catchall for everything else; 3rd party calls, external services, and utility methods.

This project will use method 2, because frankly it's a little easier to grok for small projects, however 1. is nice for larger projects due to the isolation of each domain.

Let's start by creating a few folders in src. Make a routes folder and a models folder, their contents should be fairly self explanatory, but in case you wanted assurance, route handling will go in routes and database models will go in models.

A great place to start is with the model to store the data. So create a file named secret.rs in models. Because of how rust modules work, we'll also need to add models module to lib.rs and main.rs -- add the following line to both:

pub mod models;

We'll also need a mod.rs file in the models/ folder, so throw that in there with the following line:

pub mod secret;

Now for the good stuff. In secret.rs, add this:

use std::str::{from_utf8, Utf8Error};

// 1
use aes_gcm::{
    aead::{Aead, KeyInit, OsRng},
    Aes256Gcm,
    Nonce,
};
// 2
use argon2::{
    password_hash::{PasswordHash, PasswordHasher, SaltString},
    Argon2,
};
use chrono::{DateTime, Utc};
use serde::Serialize;
use sqlx::{FromRow, Pool, Postgres};
use uuid::Uuid;

// 3
#[derive(FromRow, Debug)]
pub struct Secret {
    pub secret_id: Uuid,
    pub encoded_secret: String,
    pub salt: String,
    pub created_at: DateTime<Utc>,
    pub expires_at: Option<DateTime<Utc>>,
}

// 4
#[derive(Debug)]
pub enum SecretError {
    HashingError,
    SqlError(sqlx::Error),
    DecodingError(Utf8Error),
}

impl Secret {
    pub async fn insert_secret(
        pool: &Pool<Postgres>,
        secret: &str,
        code: &str,
        expires_at: DateTime<Utc>,
    ) -> Result<Secret, SecretError> {
        // 5
        let salt = SaltString::generate(&mut OsRng);

        let password_hash = Argon2::default()
            .hash_password(code.as_bytes(), &salt)
            .map_err(|_e| SecretError::HashingError)?
            .to_string();

        // 6
        let parsed_hash =
            PasswordHash::new(&password_hash).map_err(|_e| SecretError::HashingError)?;

        let cipher = Aes256Gcm::new_from_slice(&parsed_hash.hash.unwrap().as_bytes())
            .map_err(|_e| SecretError::HashingError)?;
        let nonce = Nonce::from_slice(b"012345678912");

        // 7
        let ciphertext = cipher
            .encrypt(nonce, secret.as_bytes())
            .map_err(|_e| SecretError::HashingError)?;

        // 8
        sqlx::query_as(
            "
            INSERT INTO secrets (encoded_secret, salt, expires_at)
            VALUES ($1, $2, $3)
            RETURNING secret_id, encoded_secret, salt, created_at, expires_at",
        )
        .bind(&base64::encode(ciphertext))
        .bind(&salt.to_string())
        .bind(&expires_at)
        .fetch_one(pool)
        .await
        .map_err(|e| SecretError::SqlError(e))
    }
}

1. We'll be using AES encryption to store the secrets. The implementation of which is FAR beyond the scope of this tutorial, just know that, for our purposes, password + secret -> encoded text.

2. In order to make guessing passwords a bit harder, we'll be using Argon2id. This means that if the database were to be compromised, it would be harder to brute force the secrets because it takes a long (comparatively) time to run Argon2id.

3. This is the struct representation of the data in our database. It's important to note here that we derive FromRow from the sqlx package. I also like to add Debug to my structs so they're easier to, well, debug!

4. In order to handle errors in a more concise way, i.e. not using a Box<dyn Error>, we'll set up an enum with a few values that can wrap the errors we might occur while we're inserting and later fetching data from the database.

5. We generate a salt here to use in our hashing.

6. The password hash is generated from the salt and the password, which we refer to as "code."

7. The password hash is used as one of the inputs for the AES algorithm along with the secret.

8. Finally we take the cipher text output from the AES algorithm, base64 encode it, and insert it into our database. We also return the full Secret object to the caller.

You might get some errors at this point -- some of these libraries aren't included in our Cargo.toml yet. Update your [dependencies] to look like the following:

[dependencies]
aes-gcm = "0.10.1"
argon2 = "0.4.1"
base64 = "0.13.0"
chrono = { version = "0.4.22", features = ["serde"] }
env_logger = "0.7.1"
log = "0.4"
rand = "0.8.5"
serde = "1.0.145"
serde_json = "1.0.86"
sqlx = { version = "0.6", features = ["runtime-tokio-native-tls", "postgres", "chrono", "uuid"] }
thruster = { version = "1.3.0", features = ["hyper_server"] }
tokio = { version = "1.6", features = ["rt", "rt-multi-thread", "macros"] }
uuid = { version = "1.2.1", features = ["v4", "serde"] }

We now need to add the route so that we can take the incoming request and call the function we just created. First, create a new folder, src/controllers. Then add two files in that folder, mod.rs and secrets.rs. We need to add that module to both main.rs and lib.rs by adding the line pub mod controllers; to each.

We also need to add the secrets module to the controllers/mod.rs -- pub mod secrets;. Now we can add the route! Make your secrets controller look like this:

use chrono::{DateTime, Utc};
use rand::{distributions::Alphanumeric, thread_rng, Rng};
use serde::{Deserialize, Serialize};
use thruster::{
    context::context_ext::ContextExt,
    errors::{ErrorSet, ThrusterError},
    middleware_fn, MiddlewareNext, MiddlewareResult,
};
use uuid::Uuid;

use crate::{app::Ctx, models::secret::Secret};

// 1
#[derive(Serialize, Deserialize)]
pub struct CreateSecretReq {
    pub secret: String,
    pub expires_at: DateTime<Utc>,
}

#[derive(Serialize, Deserialize)]
pub struct CreateSecretRes {
    pub code: String,
    pub id: String,
}

#[middleware_fn]
pub async fn create_secret(mut context: Ctx, _next: MiddlewareNext<Ctx>) -> MiddlewareResult<Ctx> {
    // 2
    let req = context.get_json::<CreateSecretReq>().await.map_err(|_e| {
        ThrusterError::parsing_error(
            Ctx::new_without_request(context.extra.clone()),
            "Bad request",
        )
    })?;

    // 3
    let code: String = thread_rng()
        .sample_iter(&Alphanumeric)
        .take(24)
        .map(char::from)
        .collect();

    // 4
    let secret = Secret::insert_secret(&context.extra.pool, &req.secret, &code, req.expires_at)
        .await
        .map_err(|e| {
            log::error!(
                "Received an error while creating a secret. Silently failing: {:#?}",
                e
            );
            ThrusterError::generic_error(Ctx::new_without_request(context.extra.clone()))
        })?;

    // 5
    context
        .json(&CreateSecretRes {
            code,
            id: secret.secret_id.to_string(),
        })
        .expect("Could not create JSON from known entities");

    Ok(context)
}

1. First we create the incoming request struct. I like to add both the Serialize and Deserialize trait regardless of which direction the middleware will actually use the struct. This way we can use the same struct to generate requests in our testing!

2. Use the json method to parse the body of the context into json. Because bodies might be large amounts of streaming data, this is an asynchronous function, so we await it.

3. In order to make a password, we generate a short random string. This will be used along with the ID to decode and fetch the secret respectively.

4. We use the function we just created to insert the secret into our database.

5. Finally, we call the json method on the context, which will serialize the passed in object and set the body as such. This will also set the Content-Type header to be application/json.

We're almost done! We just need to actually add the route to our app router. In order to do this, open up src/app.rs and the following line:

        .get("/hello", m![hello]) // This should already be here
        .post("/secrets", m![create_secret]) // This is the new line

Note that you'll have to add the create_secret import. I trust you, you can do this.

Now if you run the code (make sure your database is running!) using cargo run, you should be able to make a curl to insert a code into the database!

$ curl -X POST -d '{"secret":"test message","expires_at":"2023-02-10T17:37:09.295Z"}' http://localhost:4321/secrets  
{"code":"tEUP3oFP8vUpvZW00Zrjd1qw","id":"0398567c-a172-47e1-bceb-6e46429bf5c4"}

Assuming you got a code, you're all good! Keep on going when you're ready to do some fetchin'!