Serialized Gedanken

Fundamental estimation problems

2024-05-19T00:00:00Z

There are so many articles about the topic of estimations in software-development, I can't even count anymore how many of them I read.

Even though, almost everyone still estimates effort or time by breaking down work into tasks and then assigning a single number to each of them.
It's really beyond me why this is still the standard. Maybe because methodologies like Scrum and tools like Jira encourage it?

But to understand why this is such a big problem, a fundamental problem of estimations needs to be understood first, not only by developers but also by managers and other stakeholders.

Estimations

Let's get one thing out of the way: estimations are generally important and useful. They improve decision-making about prioritization, timing, deadlines and resource-planning.

Well, at least if everyone works together in good faith. The estimation process can also be abused and become a form of (corporate) power-play.
In those cases, there's no point in discussing about the how, so I'm going to focus on the happy-case here.

Single number estimations

I said that it's a problem to estimate work by assigning just a single number per task. Even if it's t-shirt-size estimations.

Why is that?

Imagine you are a manager (or maybe you just are one) and your developers are always giving you precise estimates like 5 days or 3 months of work for a bunch of tasks or a project.
And then they always deliver on time. That sounds great right? The developers seem to know what they are doing. Makes it easy to rely on them and plan things.

Maybe surprisingly, this is actually not good. Here is how reality usually looks like:

Developers estimates are...	Behind the scenes...
Almost always accurate	1. The developers are adding way too much "buffer time" or 2. The developers are incompetent and repeat work instead of automating it
Almost always too low	Communication issues, incompetent management (often: top down pressure)
Almost always too high	Does not happen, your data is wrong :-)
50% too high and 50% too low	🦄 You found a unicorn

Okay, fine, it's a bit exaggerated - but maybe less than you think.

Estimations are no divinations

A common misunderstanding in the field of software-development is to think that with experience estimations must get better.
And it sounds logical at first because that's how it works in other fields: someone who has to build a house for the first time will have a hard time to estimate how long the whole thing takes. But after building houses for decades, their estimations will eventually become spot on.

Unfortunately, software-development is one of the very few professions where this is inherently different.
Software-development is ultimately the process of creating automations. Every piece of work that has to be repeated over and over again will sooner or later be automated.
Once that has happened, it can be used or produced so quickly that it is not even worth to include it into the estimation process anymore.

In consequence, the only tasks that actually make it into the estimation process are those that have not been automated yet - because they have not been built in this way so far. Thinking of it from that perspective, it's not a surprise why those estimations are comparably difficult.

But it's not only that. There are also many factors that simply cannot be estimated precisely when starting. For instance, a technical dependency that is required for the task at hand has an unexpected bug that needs to be worked around.
All in all, even if we try hard to estimate a task as precisely as possible, we inevitably end up with a probability distribution; and most likely a rather complicated one.
And since there are many factors which all vary in their own way, it is not unreasonable to assume that the outcome will often follow something that is not far from a normal distribution.
So it might actually look something like this:

The graph shows how likely it is (in percent) that the task will be finished in a given amount of time on the X-axis (in days or weeks or months - it doesn't matter here).

To come back to the table from the beginning - what would the developer give as a single number based on that distribution?
Due to the symmetry it clearly looks like a 4 on average.
However, the chance for the task to be finished after exactly 4 days is extremely low; it will sometimes be a bit more and sometimes a bit less, maybe 3 or 5 days.
One day difference might not sound like a problem but what if those are months not days? What if it's a law that we are supposed to follow with a given deadline?

Looking at it from another perspective: when using such a technique for estimations to decide on deadline dates, half of all deadlines will be missed.

By now it should be clear why single-number-estimations are very limited.
But I will add another example to make it even more obvious.

Let's imagine there is a complicated feature that needs to be implemented by a single developer Alice.
The feature requires interoperation between multiple systems, including a system that Alice is unfamiliar with.
However, there is another developer, Bob, who knows all about the system and can share that knowledge to decrease development time drastically.
But Bob is busy with another urgent and critical project and it is yet unclear if he will be able to provide any support at all.

How does the probability distribution look like in this case?

There are two peaks, one at a low effort (~4 days) when receiving a bit of help by the other developer and one without their help (~12 days).

Just estimating the mean value here is even more misleading than in the previous example. The mean of this distribution here is 7.2, but when looking at the graph we can see that the chance of the outcome to actually be 7.2 or even close to it is pretty much zero.

Using the mean alone is obviously a terrible idea.

Unfortunately it's unrealistic to model the effort with a precise mathematical distribution like above for every single task, let alone projects that consist of hundreds or thousands of subtasks.

Tackling the shortcomings

In reality, simple mean values are mostly used when doing estimations.
And since they are lacking, how do the estimating developers usually attempt to work around this problem in various ways?
Let's look at a couple of the common ones:

Buffers

After being yelled at by their manager for a missed deadline, "buffers" will be added to the estimation values. This is very common.
Doing so is usually based on gut-feeling, or at least I've never seen a guideline for it or even a standard.
Sometimes the deadline is communicated through multiple layers and every person adds their own buffer to protect themselves until all meaning is lost.

All of that does not help in cases like the Alice/Bob example. Even fairly big buffers possibly won't really help if the second probability-peak isn't covered.

In the end, buffers lead to suboptimal planning and resource allocation and for example significantly increase the time-to-market.

Splitting tasks

Another classic. Once it is understood that single-number estimations don't work well, the size or scale of the estimated tasks is made out to be the root of the problem.
Those tasks are just all too big and can't be estimated precisely! Hence, tasks that are too big are must be split into a smaller tasks and so on until the estimation for every task is below a certain threshold.

There are indeed some benefits in splitting big tasks, it's not like this is a bad idea in general. But it comes with problems. Estimation is a time intense process and the more granular the tasks become the more friction and bureaucratic process is added to manage them.
In addition, the effort to actually combine the tasks into an end result is often missed or underestimated, making it so that the sum of the child tasks does not match the whole project.
Finally, sometimes it's just not possible to meaningful split tasks further.

Ultimately, it is not a solution to the single-number estimation problem; though it can aid in other ways, such as improving the common understanding of every task within the team.

Side channels

There are more workarounds, but the last one I'd like to mention here is the communication via other channels.
If tools or processes force to enter single number, people sometimes step up and communicate that the estimation might be value X but that the confidence into that estimation is low.
This is a reasonable approach and better than nothing, but it is also unstructured and unsuitable as a professional solution in general.

Confidentiality intervals

My preferred practical but effective solution to improve the situation without making it too complicated is the use of confidentiality intervals.

The idea is to describe two numbers instead of one: the lower and upper bound which capture 90% of the expected outcomes. For example, if the interval is defined as [2, 5] and we measure in days, then that would mean that the chance is 90% that the task will take between 2 and 5 days. And only 10% chance that it will take less than 2 or more than 5 days.

It doesn't really matter what percentage we choose - 90% or maybe 80% or 75% are all fine. What matters is that we have introduced a way to indicate the trust we have in the precision of the estimation.

(Note: the classical way would be to use the mean and the variance or standard derivation to describe a probability function. However confidence intervals are a bit more intuitive in their usage when it comes to most estimation processes)

How to use confidentiality intervals

1.) We decide on a timeframe and ask the developer to estimate the probability that the feature will be completed within that timeframe.

2.) We ask the developer what the deadline would have to be so that it will only be missed with a chance of 10% (or some other percentage)

Approach 1.) is useful in cases where a timeframe is set by a 3rd party and we can't control it.
For instance, a legal requirement that needs to be complied with by a certain date.
In that case, we can gain information about how likely it is that the deadline won't be met. And by knowing that, we can react to it, for example by assigning more resources or preparing for the consequences of missing the deadline.

Approach 2.) is more useful in cases where we can control the deadline. We might want to release a feature and we need to communicate it in advance, for instance so that the marketing department can prepare, book and release and advertisement campaign.
In this case we can decide on how much of a problem a delay would be and in consequence set the appropriate confidentiality level. We then ask the developer to tell us about the deadline.

Bonus

I hope that I could shed some light into why there are often problems with estimations in software-development and also how to improve on it - for both technical and non-technical folks.

If you made it that far, here's another question: why is usually only the time/effort estimated that is required to complete a task or project, but not the time/effort to maintain things?

I believe that this is one of the reasons why companies or teams eventually get stuck and become unable to deliver anything new. But this is for another article. :-)

Should you have solved this mystery then please let me know in the comments. Also, I would be happy to learn if you found an even better practical solution for estimation than confidence intervals.

SQL as API

2023-12-25T00:00:00Z

SQL in the API???

I know what you are thinking: Exposing an API that accepts SQL is crazy. It's a terrible idea.
Especially if the API is exposed on the internet. Doing that is insecure and will lead to SQL injection attacks, it is a nightmare to maintain and it will lock the backend implementation into a specific technology (some ANSI SQL database).

But is that really true? Time to re-evaluate!

There is always an exception

Engineering is about making trade-offs. There is no one fits it all solution. Therefore, here I'll present the kind of scenario in which SQL is in fact the best choice to make a backend API work out.

But let's take a step back first. Many APIs can be simple. Often, a CRUD API that allows for fetching, creating, updating and deleting entities by some ID is all that is needed to get the job done.
No arguing with that. And there is no reason to over-engineer things if this is really the case.

On the other hand, there is a considerable number of applications where the business will want to expand its features in a way that forces the API to become more and more complex over time.

Even just a basic web-shop can be a sufficient example for that kind of scenario. Let's say there is this web-shop which sells bicycles. Having a classical CRUD API already gets us very far in the sense that we can add, delete and update items (bicycles). We can also list them and show them to a user. But what about allowing the user to filter the various bicycles by their properties? Like color, size, price, etc.

Sounds simple at first. Just add some filters-field to the request. In the HTTP / REST world it might like look this:
GET /bicycles?color=red&size=large&price=1000-2000
Problem solved.

But faster than you can imagine, the next requirement comes in: OR filters!
Maybe they should be combined with AND filters so that we can express constraints like bicycles that are (made of steel AND weigh from 10 to 20kg) OR (made of carbon AND weigh from 5 to 10kg).
Dealing with unknown values (typically represented by null in databases) and other subtleties will quickly creep into the codebase as well.

And before long there is some complex, custom-made spaghetti-ball of filter logic, even for a simple shop website. Sounds familiar? If not, lucky you. :-)

Usually, at some point, developers start to structure those filters and start to use e.g. json for more complicated requests. It might look like this:

{
  "filters": {
    "or": [
      {
        "and": [
          {
            "property": "material",
            "operator": "equals",
            "value": "steel"
          },
          {
            "property": "weight",
            "operator": "between",
            "value": [10, 20]
          }
        ]
      },
      {
        "and": [
          {
            "property": "material",
            "operator": "equals",
            "value": "carbon"
          },
          {
            "property": "weight",
            "operator": "between",
            "value": [5, 10]
          }
        ]
      }
    ]
  }
}

At the latest at that point your nose should notify you that this is a design smell.
Why? Because we are essentially inventing our own query language here. And inventing a custom language should be pretty much the last resort, I hope we agree on that.

SQL to the rescue?

Since this article is about SQL in APIs, how does SQL help us here? We surely don't want to have the webshop-frontend send database queries to the backend to get the results.
Well, to be honest, that already has become a thing nowadays. Hasura and alternatives are doing pretty much that - they generate a fully fledged graphql API based on the SQL schema of the database.

But besides being pricey and requiring specific database technology and locking you into that technology and having limited customizability, there is also just a much more lightweight alternative: Exposing a subset of SQL in your API.

The beauty of doing so is threefold:

SQL is well-known language with plenty of documentation to learn it as well as support for it in almost every programming language. No need to write your own SQL parser or anything, whereas with your "own" language you need to write one. (Actually: it doesn't even have to be SQL, as long as it is a well understood and standardized query language. But SQL is probably the best choice in most cases)
We can expose subsets of SQL and add more and more functionality over time and when needed, to fulfill the business requirements - without breaking existing queries and tooling.
We remove a whole layer of confusion because of conversions from and to SQL (if the backend uses an SQL database anyways). Also, SQL is compact and easy to read for everyone.

But isn't that unsafe? Don't we execute SQL provided by the frontend directly against the database? That sounds like a security nightmare. And indeed, people have tried that and it has gone horribly wrong. So don't do that!

Instead, for such a system to work properly, we have to do the same that we do for any other API:
validate the input and sanitize it before executing it. And by that I'm talking about a white-list approach.
We only accept SQL from the request that we specifically allow. Everything else is rejected.

Mind that we don't do that by validating the SQL string! We will first parse the SQL string into a data structure, then validate this structure and finally convert the structure back into SQL.
In other words: the SQL executed against the database might look a little different than the one we received from the API.
And this is good. It allows us to do modifications to the SQL structure if necessary (such as adding constraints and transforming identifiers like column names) and make our API even work with non-SQL databases!
Yes, you heard it right: using SQL in the API doesn't mean we also need to actually run it against an (ANSI-) SQL database. We could choose to generate an elasticsearch query instead, for example.

Okay, enough talk about concepts. How does this look in practice in terms of actual code?

I hope for you, dear reader, that you are having the luxury of working with a programming language that allows to model ADTs in a nice way and offers a builtin technique such as pattern matching for decomposition. If you don't... well, I suggest to learn a few new languages that do support those features, it's 2023 after all. (sorry to all Go developers out there)
I'm going to use Scala for the following code examples but you should be able to understand the code without knowing Scala.

Defining the data structure

Let's stick to our bicycle example above. To keep it simple, we will only want to use SQL for the filtering part.

How would such an SQL filter look like? As a full SQL query it would look like that:

SELECT *
FROM bicycles
WHERE    (material = 'steel'  AND weight BETWEEN 10 AND 20)
      OR (material = 'carbon' AND weight BETWEEN 5  AND 10)

What we care about is the where-expression only:

(material = 'steel' AND weight BETWEEN 10 AND 20) OR (material = 'carbon' AND weight BETWEEN 5 AND 10)

In our code we typically model such a structure with a recursive tree-structure:

sealed trait Expr
  case class Column(name: String) extends Expr
  case class And(left: Expr, right: Expr) extends Expr
  case class Or(left: Expr, right: Expr) extends Expr
  case class Between(expr: Expr, lower: IntegerValue, upper: IntegerValue) extends Expr
  case class Parenthesis(expr: Expr) extends Expr
  case class Equals(column: Column, value: Value) extends Expr
  case class ValueExpr(value: Value) extends Expr
  // ... more expressions here for other SQL features

sealed trait Value
  case class StringValue(value: String) extends Value
  case class IntegerValue(value: Int) extends Value
  // ... more value types here for other types of values in SQL

That is all we need to describe the filter above - and other even more complicated filters as well.
Usually, we will not define the above data structure by ourselves. We should use a library that already gives us those definitions.

Processing the SQL

Such a library will also help us to parse the SQL, so all we really have to do is this:

val testSqlStringFromAPI: String =
  "(material = 'steel' AND weight BETWEEN 10 AND 20) OR (material = 'carbon' AND weight BETWEEN 5 AND 10)"

val filterExpression: Either[Error, Expr] =
  sql_library.parse(testSqlStringFromAPI)

filterExpression match {
  case Left(error) =>
    println(s"The SQL was invalid!") // return error 400 to the frontend here
  case Right(sqlExpression: Expr) =>
    ??? // This is where it gets interesting! Process the SQL here
}

The last thing we have to do now is to decide, which features of SQL we want to allow in our API and also what we want to generate based on the given SQL. In the easiest case, we actually are working with an SQL database. Then we can just generate SQL again. If we were to work against an elasticsearch instance, we would generate an ES query instead.

Let's assume for now, that we are not allowing or-clauses in our API yet.
And to keep the example small, we will just recreate the SQL as a string. In reality, we would of course build the SQL using the same library again.

In the following code we traverse the SQL structure and validate/transform it in the way we need:

val columns = List("material", "weight", "color") // Some columns we use/allow in the API

// Recursive function to traverse the structure
def processSqlExpr(expr: Expr): String = expr match {
    case Column(name) =>
      if(columns.contains(name))
        name
      else
        throw new Exception(s"Column $name is unknown and not supported!")
    case And(left, right) =>
      s"(${processSqlExpr(left)} and ${processSqlExpr(right)})"
    case Or(left, right) =>
      throw new Exception("Or-clauses are not supported yet!")
    case Between(expr, lower, upper) =>
      s"${processSqlExpr(expr)} between ${processSqlValue(lower)} and ${processSqlValue(upper)}"
    // the following one removes double parenthesis! :-)
    case Parenthesis(Parenthesis(expr)) =>
      s"(${processSqlExpr(expr)})"
    case Parenthesis(expr) =>
      s"(${processSqlExpr(expr)})"
    case Equals(column, value) =>
      s"${column.name} = ${processValue(value)}"
    case ValueExpr(value) =>
      processSqlValue(value)
  }

def processSqlValue(value: Value) = value match {
  case StringValue(value) =>
    s"'$value'"
  case IntegerValue(value) =>
    value.toString
}

That is all we need to handle arbitrary complex filters. Let's try it out:

processSqlExpr("(material = 'steel' AND weight BETWEEN 10 AND 20) OR (material = 'carbon' AND weight BETWEEN 5 AND 10)")
// Error: Or-clauses are not supported yet!

processSqlExpr("((material = 'steel' AND weight BETWEEN 10 AND 20))")
// Output: (material = 'steel' and weight between 10 and 20)
// ^^^ Note how the spelling of the SQL is slightly different from the input and that we removed the double parenthesis.

If we later want to add support for OR-clauses, it's as simple as changing

throw new Exception("Or-clauses are not supported yet!")

into

s"(${processSqlExpr(left)} or ${processSqlExpr(right)})"

In fact, we now have the ability to expose pretty much every SQL feature to our API users with almost no effort.

Wrapping up

Since SQL has more features and syntax than shown in the example, processSqlExpr() will be a bit longer, since it has to handle more cases (or alternatively have a catch-all case that throws an error if an unsupported feature is used).

As you can hopefully see here, this approach is very different from any kind of ancient string-manipulation of SQL, which is despised for good reasons.
Since in reality we would not really create SQL strings but build the SQL using a library, there is also no way that we would accidentally create invalid SQL or SQL that is insecure and allowing for SQL injections (e.g. by forgetting to escape strings or removing comments from the SQL).

If we later want to enable more features of SQLs power, we can easily do that: and existing queries will keep working. And that also includes more complicated features like different kinds of pagination, limits, ordering, grouping, etc.

Also note that we can still change the database-schema without problems, as long as we keep the "column-names" that we expose in the API the same. We will just have to map them to the new schema during our processing.

Hold on!

By now you should have a good mental model about how that approach can play out in technical terms. But it is important to understand that it is still a niche solution. It comes with some drawbacks that better be well understood!

Documentation and error-handling

The first one is documentation. While SQL is widely understood in general, we commonly only need a subset of SQL in the API. Which subset exactly and the restrictions that apply need to be documented well, otherwise it won't be clear from the API endpoints how the API can be used.
This will be less of a problem if there is only one or a few selected consumers of the API, but it's different if it is a public API or even a free public API. For the latter types of API, this approach is probably not a great solution.

On top of that, if wrong/invalid SQL is provided, the error messages provided by the backend should be helpful.
Since the request will usually contain the SQL as a string, there is no additional structure, which means that there is little tooling support for the one who makes the request.

On the good side, reading and parsing SQL is usually easier than reading and parsing big json structures. And SQL can easily be copied into an sql editor to get formatting and syntax support if needed.

Performance considerations

Parsing the SQL can be simple and if done right will probably have only a negligible impact on the performance of the API. But for cases where latency is critical or the mass of requests is so high that every last drop of performance matters, SQL should probably not be used.

Also, just like it is a problem for graphql APIs, the possibility of DDoS attacks needs to be considered. Since SQL (in the way implemented above) allows arbitrary nesting by default, without any countermeasures (such as limiting nesting levels or the number of conditions) it can easily be abused to cause high load on the server with just a small number of requests.

Vendor lock-in

Even though SQL is a standard, if database specific features or functions can are exposed in the API, it will be harder to switch to a different database technology on the backend later.
However, this also applies to a non-SQL kind of API in the same way, so it's not strictly a drawback of SQL in the API. The difference is though that it might be tempting to allow more SQL features if the API already uses SQL and it's easy to add them.

Last words

Usually, I would expect that most APIs don't require the flexibility that SQL offers. And using SQL has been despised by people for the longest time, probably for good reasons at the time.
On the other hand, not using a solution that is as powerful and flexible as SQL is a recipe to reinvent the wheel and create a lot of accidental complexity in the code base if the flexibility is required by the API users.
And with the techniques shown, it is safe and convenient to make use of its power.

I hope this article has convinced you not to despise such a solution by default, but weight he pros and cons carefully when you have a situation at hand where the API is considerable complex.

Scala's great ecosystem

2023-12-22T00:00:00Z

Nothing ever work... mostly

I often feel I want to rant about how "nothing ever works" in tech.
But today is one of those rare days when I need to shout out to those people in tech who make my life so much easier: the creators of Scala and especially its great ecosystem, in particular ZIO, caliban and http4s.

The challenge

Recently I've been working on a web application for a client.
This application is a classical analytics tool that displays charts and tables.
What makes it stand out a bit from similar tools is the number of ways in which the user can explore the data, including selecting and combing various filters dynamically, applying grouping, time-ranges, drilling down, etc.
Furthermore, users can only see specific data depending on their roles, so there is no way the application can allow the user access to the underlying data directly (such as tools like tableau or looker do) which would makes things much simpler.

In consequence, there is a backend system with a graphql API that allows the frontend to request the different kinds of data (called "metrics") that the frontend needs and in which shape it needs them in.
For better or worse, the backend then takes such a request and smartly generates SQL queries which are executed by a Postgres database.

The SQL queries that are generated depend on the request. But one thing is important: for performance reasons one request will contain multiple metrics.
And because similar metrics can be resolved by a single SQL query (e.g. because they use the same table), they will be grouped together in the SQL to improve performance and reduce database load.
Within the backend (but transparent to the frontend) the structure looks something like that:

[metric1, metric2, metric4] -> sql1
[metric3, metric6]          -> sql2
[metric5, metric7, metric8] -> sql3
...

In other words, multiple SQL queries will be executed for a single graphql request.

Experienced developers can immediately tell what this means: if there are 10 SQL queries generated and 9 of them resolve really fast but the last query is slow, then this one slow query will delay everything since it has to be completed before the graphql request can be responded to by the backend.
This leads to a bad UX where the user waits for all their results for a long time, even though they could already see 90% of their requested numbers while waiting for the last 10%.

This problem had to be tackled.

Graphql subscriptions to the rescue

There are certainly different ways to deal with that problem, including scaling up the database or even changing database technology completely.
However, if it were just possible to return the already finished partial results to the frontend, that would already improve the UX drastically with little effort.

Luckily I'm not the first one to deal with this kind of problem. Graphql has a solution builtin: subscriptions.
It allows the backend to use websockets for asynchronous 2-way communication between backend and frontend. And that also means the backend can send data to the frontend in a streaming way. Just what I needed.

Since I had never used websockets before, I was wondering how difficult it would be to get the backend to support it.
To do so, three parts where necessary:

1.) I had to rewrite the backend code that was creating and executing the SQL and then turning it into a proper response for the frontend.
This code was already written with ZIO in an async way, but it just awaited all SQL queries to finish before processing the results.
This had to be turned into a streaming approach using ZIO Streams so that already finished SQL queries would be processed and returned to the frontend immediately without waiting for the other SQL queries.

2.) The graphql API generated by caliban had to be extended to support this kind of subscription. This essentially meant adapting the existing query and turning it into a streaming subscription by using the logic from step 1.).

3.) The webserver configuration had to be changed to support websockets for the new graphql subscription and be made work together with the graphql library.
Of course existing features for regular http-requests such as authentication, logging, metrics etc. should also work for websockets.

The happy surprise

When it comes to dealing with a new technology that is rather niche while having to get multiple libraries to talk to each other and work together, I'm usually quite skeptical. Those tasks often become a time sink.

That's why I was very positively surprised when I managed to finish all 3 steps in just half a day. And that included having to downgrade the version of http4s to the latest stable version (instead of the previously the "bleeding edge" milestone releases).
This forced a bigger refactoring, but it allowed me to use an integration between caliban and http4s so that I only had to write a few lines of code to get the websockets to work. Thanks to the caliban folks for providing that one!

I then spent the rest of the day trying to get our already setup graphiql to work with new websockets API... but that's a different story.

For me, two things are really remarkable here.

First, Scala's powerful type-system is often disregarded as academic and too complex with little practical benefit.
But here it shows it's power, because it enables such a great ecosystem in which libraries that know nothing of each other can be made to work together with just one line of imports and sometimes a few extra lines for configuration or a small separate dependency (which can be provided by a totally separate party).
This is often missed by people: the type of a programming language has a huge impact on the quality of its ecosystem.

Second, I find it amazing that I can rewrite a crucial piece of the application logic to work in a lazily-streaming way, downgrade the http library, add a new project dependency and integrate it into the code, glue things together, then fix the millions of compile errors and then... it just works and runs all smoothly.
There was no manual or automated testing or other kind of trial&error during the whole process - just fixing compile errors and then on the first run it all worked exactly as expected and the data was streamed lazily just like I planned (which I could confirm after being able to fix graphiql).

And while it's certainly not hard to set up some graphql subscription POC in almost any language / ecosystem from scratch, it is a whole different story to do that within an existing application and having to apply quite the refactoring for it while not breaking anything in the process.

So for me, the Scala way of doing things is not only a huge time saver and makes me super productive.
It also just feels much better than running and fixing tests one by one or even running the application and testing it manually, just to find that something breaks at runtime.
Even just the pleasure of this way of working makes learning some harder concepts of the Scala language more than worth it.

Context-dependant code is better than its reputation

2023-02-11T00:00:00Z

Should a programming language allow to overload operators? Should it allow to use symbols for method-names and variables? And is it okay if the same piece of code behaves differently in different places, depending on its surroundings/context?
Ask those questions and enjoy a never-ending flame war among software developers.

Contexts outside of programming

To get right to the point: I believe it is a good thing if a language allows to have (explicit) contexts which can change semantics of code and give meaning to symbols and even "overload" them. My reasoning is that we, as humans, are used to act differently depending on our context all the time. And we can take advatange of this built-in optimization of our brains to increase productivity.

We already do that outside of programming. Uniforms are an example. They don't just allow everyone to recognize someone's function or position; they also remind the wearer of the context they are currently operating in. A judge is reminded that they are now supposed to act as a judge and not just as themselves, reinforcing fair decision-making.

Something that maybe hits closer to home are pyjamas. At least for me, when wearing them I get into a very different mood than when wearing stiff and formal clothes. It makes it easier for me to fall asleep as well. On the other hand, working in pyjama feels just wrong. Music and scents are yet another feature that can have similar impacts. And the list goes on.

Contexts in programming

So how is this relevant in programming? Well, when programming we do two things. We express our intent/thoughts as code in a way that 1.) the computer understands it but also 2.) ourselves and others can understand it later. The more concise and precise we describe the real thing the easier it is to understand and change the code.

In the real world there are many symbols have different meaning depending on the context.

For instance, what does 5 ± 2 mean? Without context that's hard to tell. Does it mean 3 or 7? Does it mean 5 with a tolerance of 2 or does it maybe mean A mean of 5 with a deviation of 2? The plus-minus sign has many meanings, depending on the context.

The same is true for many other domains, not just mathematical notations, but let's stick to that for now as an easy to understand example.

Of course, as developers we don't necessarily have to follow all conventions and notations of the real world. And sometimes we deliberately choose not to. But being able to stay very close to business-lingo in our code can definitely be very helpful. It reduces friction when encoding business requirements into code and can make code more concise in general.

Usage in practice

How does it look like if we try to make our code be context dependant?
Most programming languages actually don't support this or at least make it very hard and unergonomic. But there are some languages that are great for it. Scala is such a language. It embraces contexts and makes them ergonomic to use while making it hard to shoot yourself into the foot.

So for the following real-life examples, we're going to use Scala.

Using ± to describe choices

Let's say we are dealing with a business domain where we often have to deal with choices and our business experts use the ± in all their technical documents.
How can we make our code match the domain?

First we define what a choice is

enum Choice:
  case Numbers(lower: Int, upper: Int)
  case Choices(lower: Choice, upper: Choice)

A choice can be 2 numbers (e.g. 5 ± 2 would be 3 for lower or 7 for upper).
But if we have a choice and work with it, we might end up with a choice of choices, e.g. ((5 ± 2) ± 1) would be either 3 ± 1 or 7 ± 1.

We are working with regular integer values (Int) from the standard library. In our code we want to write 5 ± 2 but the type Int is already defined. Luckily Scala supports statically typed extension methods:

extension (number: Int)
  def ±(diff: Int): Choice.Numbers = Choice.Numbers(
    lower = number - diff,
    upper = number + diff
  )

This already let's us to write code that is valid and works as expected:

print(5 ± 2) // prints: Numbers(3,7)

To make it a bit more interesting we also want to work on choices of choices, i.e. calling ± not only on numbers but on choices themselves. To allow that, we add another method:

extension (choice: Choice)
  def ±(diff: Int): Choice.Choices = choice match
    case Choice.Numbers(lower, upper) => Choice.Choices(
      lower = lower ± diff,
      upper = upper ± diff
    )
    case Choice.Choices(lower, upper) => Choice.Choices(
      lower = lower ± diff,
      upper = upper ± diff
    )

We could define this code top-level and just start using it in our project - but since we want to create an explicit business context, we move the whole logic into its own "context" object (or module):

object choiceContext {
  extension (number: Int)
    ...

  extension (choice: Choice)
    ...
}

The encapsulating object really just acts as a namespace that needs to be explicitly imported. If we just try to use the new syntax in a project then it will fail by default:

val choice = 5 ± 2
print(choice)

Fails with:

value ± is not a member of Int
   val choice = 5 ± 2

This is good because without further information what does it mean when we use ±?
We should be clear about what we mean when we use ± and we can do so by telling the compiler (and thus other developers) that we are working within the "context of choice". Then it will compile and behave as expected:

 import choiceContext._ // Make it explicit that we want to be working in the choice context

 val choice = 5 ± 2
 println(choice) // prints: Numbers(3,7)

 val moreChoice = choice ± 1
 println(moreChoice) // prints: Choices(Numbers(2,4),Numbers(6,8))

We can now work with a nice and concise mathematical notation and extend it with more functionality to match the lingo of our domain experts. Who knows, the code might get so close to what those experts are used that they can understand the source code or even write it themselves without the help of a developer. (Let me dream of it at least...)

Using ± in another context

It probably does not happen very often, but what if we require to use ± in a second context, like the context of tolerance like in physical engineering?
We might want to write code like val tolerance = (5 ± 2) and then if( 42.exceeds(tolerance) ) or similar, because that's what our domain exports do.

We can do that in the same way as before:

case class Tolerance(from: Int, to: Int)

object toleranceContext {
  extension (number: Int)
    def ±(diff: Int): Tolerance = Tolerance(
      from = number - diff,
      to = number + diff
    )
}

import toleranceContext._

print(5 ± 2) // prints: Tolerance(3, 7)

The first question that should pop up in your head is: can we use two contexts at the same time?
And sure, we can use both contexts or even more of them at the same time!

But then what happens if we mess up? In this example we use ± in both contexts - so what will happen if we call ± on a number? Will the result mauybe depend on the order of imports?
Fortunately it won't (this would be insane anyways). The compiler will refuse to compile:

import choiceContext._
import toleranceContext._

val result = 5 ± 2

Fails with:

value ± is not a member of Int.
An extension method was tried, but could not be fully constructed:

    toleranceContext.±(5)    failed with

        Reference to ± is ambiguous,
        it is both imported by import choiceContext._
        and imported subsequently by import toleranceContext._
val result = 5 ± 2

Great! At least it's hard to make any mistakes by accidentally screwing up context imports.

So what do we do if we have two contexts that just share one common symbol that we want to use at some point? Luckily we are not forced to shenanigans like splitting up our code and then using just a single context for a specific part.
Instead, we can tell the compiler explicitly to use the version of ± that we are interested in:

val result = choiceContext.±(5)(2) // a bit ugly but works

There are other ways to resolve this situation and make it more ergonomic, but let's not go down the rabbit hole here.

A word about naming

So far I think this looks all great, and I hope you agree me. But you might still be sceptical for a different reason, and rightfully so!

Using symbols as names and identifiers in code is not allowed or discouraged in many programming languages and that has a history. And while I believe it is good to be able to use them, one should do so with great care.
With great power comes great responsibility. When using a symbol that is not widely used/understood then my strong recommendation is to always create a method with a readable name and make the symbolic name just an alias.

For instance, in the case of ± I would write the following code in real life:

extension (number: Int)
  // Alias for plusminus
  def ±(diff: Int): Choice.Numbers = plusminus(diff)

  def plusminus(diff: Int): Choice.Numbers = Choice.Numbers(
    lower = number - diff,
    upper = number + diff
  )

This allows to use both 5 ± 2 as well as 5.plusminus(2) and allows to easily find a name that can be searched for or pronounced more easily.

Conclusion

It will certainly stay a subjective matter, but if a programming language offers good support and is statically typed then it becomes possible to make code much more concise and readable while keeping it safe from accidental mistakes.
In such a case I find there is very little reason to restrict the usage of symbols and context-dependant code behaviour.

Functional error-handling with stack traces

2022-12-17T00:00:00Z

Over the time I have developed a go-to approach for error-handling in my pure functional Scala projects.

I've seen developers thinking that exceptions and their benefits don't play well together with a (pure) functional programming style. But this is a wrong assumption. Functional error-handling and exceptions can be combined to get the best of both worlds and here I'll show how.
In my examples I use the ZIO library, but it works similar with Cats-Effect or other async/effect libraries.

Why exceptions?

Some developers use an error-handling approach that relies on custom defined classes/types which are unrelated to Exceptions. They return those errors using a Result type such as Scala's builtin Either. While this works, it commonly has the severe drawback of losing the ability to inspect the stack trace when debugging an error.

Having a meaningful stack trace can make the difference between reading the logs for a minute or having to dig into a problem for days. They are just too valuable not to use them, so I always use exceptions for my error-types.

I start with a common exception-class for the whole project. It gets a unique name, is easy to create and spot in logs / stack traces.

class GeneralException(message: String, cause: Throwable) extends Exception(message, cause)
object GeneralException {
  def apply(message: String, cause: Throwable = null) = new GeneralException(message, cause)
}

Using it is simple.

if(condition) {
  throw GeneralException("boom")
}

Or, in a pure functional style using ZIO as the Result type:

if(condition) {
  ZIO.fail(GeneralException("boom"))
} else {
  ZIO.succeed(42)
}

It is generally much better to define and use semantically meaningful error-types. But everyone is lazy sometimes. In those cases, using GeneralException directly is still much better than relying on Java's builtin Exception.

The next step is therefore to specific errors for all the things that can go wrong. Those help to make code easier to read and also allow to catch and handle only specific errors while passing along others.

sealed trait Error extends GeneralException

case class UserNotFound(id: UserId, cause: Throwable = null)
  extends GeneralException(s"User with id $id not found", cause) with Error

case class SendingEmailFailed(email: Email, cause: Throwable = null)
  extends GeneralException(s"Sending email to $email failed", cause) with Error

case class SendingSmsFailed(phoneNumber: PhoneNumber, cause: Throwable = null)
  extends GeneralException(s"Sending SMS to $phoneNumber failed", cause) with Error

case class SendingNotificationFailed(id: UserId, cause: Throwable = null)
  extends GeneralException(s"Notifying user with id $id failed", cause) with Error

In this example I am defining just a flat list of errors, but this can totally become a complex hierarchy with multiple common error-traits. You might have noticed that the cause parameter is set to null by default - something we usually try to avoid in the Scala world. However, in this case I make an exception for improved ergonomics. Error-handling is really one of those things that need to be as easy as possible. Otherwise, people tend to take shortcuts.

Using it in code

Here is how it looks when I write my business logic using different methods that throw the errors above.

def notifyUser(userId: UserId, message: Text) = {

  val notificationResult = for {
    user <- getUser(userId)
    notificationSentTimestamp <- if (user.preferences.prefersEmail) {
      sendEmail(user.email, message)
    } else {
      sendSms(user.phoneNumber, message)
    }
  } yield notificationSentTimestamp

  notificationResult.mapError(e => SendingNotificationFailed(userId, e))
}

I find this pure-functional monadic approach of error-handling quite elegant.

In the example, getUser might fail with a UserNotFound-error and sendEmail and sendSms might fail with SendingEmailFailed and SendingSmsFailed respectively.

The different actions are executed and if an error occurred then I turn it into a SendingNotificationFailed-error. Wrapping errors is optional. If I were to remove this line, the resulting error-type would just be a combination of all possible errors that can occur. But it's often nicer to hide the underlying errors and make the result more meaningful - and the method-signature shorter.

So what exactly is the return type of the notifyUser function?
If I keep the last line then it is:

def notifyUser(userId: UserId, message: Text): IO[SendingNotificationFailed, NotificationSentTimestamp]

If I remove it becomes:

def notifyUser(userId: UserId, message: Text): IO[UserNotFound | SendingEmailFailed | SendingSmsFailed, NotificationSentTimestamp]

If I do not add an explicit return type to the method, it will be inferred for me by the compiler, including all possible errors. That means, when adding more functionality later (e.g. additionally sending a push notification to the smartphone) or changing the error-handling, I won't have to refactor the whole chain of methods that call each other.
This is very useful, because a small change can otherwise force to refactor dozens of method signatures.

Note that IDEs like IntelliJ can still show the return type inline, so even if it is not explicitly annotated, it is still visible and can serve as documentation.

The resulting logs

Using this approach proves me with a very nice stack trace when I log an error. Most developers in the JVM world should be familiar with that.
Let's assume that my call to notifyUser failed because the SMS could not be sent. When looking into the logs, I will see something like this:

Exception in thread "main" SendingNotificationFailed: Notifying user with id xxx failed
    at MyProject.method3(MyProject.scala:19)
    at MyProject.method2(MyProject.scala:12)
    at MyProject.method1(MyProject.scala:8)
Caused by: SendingSmsFailed: Sending SMS to 123456789 failed
    at MyProject.method5(Sms.scala:143)
    at MyProject.method4(Sms.scala:77)
Caused by: ConnectionError: Could not connect to SMS provider XXX
    at org.example.SmsProvider.connect(Foo.scala:123)
    ... 2 more

And so on. It's a nice chain of errors which makes it very easy to understand both the high-level impact and the root cause.

API errors

If this is some internal process in my application then this kind of error-logging can already be good enough.

But usually I need to expose errors to a third party such as an external system or an end-user. In that case I typically don't want to expose internal technical details.
I also want to keep my API stable. Therefore, I define another layer of error-types which are describing what my API exposes.

sealed trait ApiError

case class ApiUserNotFound(id: UserId, cause: Throwable = null)
  extends GeneralException(s"The user with id $id was not found", cause) extends ApiError

case class ApiSendingNotificationFailed(id: UserId, cause: Throwable = null)
  extends GeneralException(s"Notifying user with id $id was not successfull", cause) extends ApiError

My internal errors are created at some point during the execution and before returning a response, I turn them into API errors.

def toApiError(error: Error): ApiError = error match {
  case e @ UserNotFound(id, _) => ApiUserNotFound(id, e)
  case e @ SendingNotificationFailed(id, _) => ApiSendingNotificationFailed(id, e)
  // ... and so on
}

This looks a bit redundant but has an advantage: I can now modify my internal errors in any way I want - if I modify them in a way that would impact my API then the compiler will make me aware of it because transforming internal errors into api errors will fail to compile.

Adding references and codes

On top of that, I find it useful to add two more things: error-references and error-codes.
Error-references are unique identifiers that allow me to find the error in the logs.
Error-codes are a well-defined list where each entry describe the type of error so that an external system can react to specific errors easily.

Error codes

Let's start with error codes. I'm extending each of my ApiError types with a specific error-code:

type ErrorCode = String // for readability

sealed trait ApiError

case class ApiUserNotFound(id: UserId, cause: Throwable = null, errorCode: ErrorCode = "EC_UserNotFound")
  extends GeneralException(s"The user with id $id was not found", cause) extends ApiError

case class ApiSendingNotificationFailed(id: UserId, cause: Throwable = null, errorCode: ErrorCode = "EC_NotificationError")
  extends GeneralException(s"Notifying user with id $id was not successfull", cause) extends ApiError

I have used different formats here - numbers, strings, whatever fits my needs, but if I have the choice I prefer to add a prefix and use easy to read names.
To keep the example simple I'm using strings and default-arguments here, but optimally an enum is used here to ensure that error-codes are never reused by accident.
The ValueEnum of Enumeratum can be used to do so.

Error references

Error-codes are nice to indicate the type of error, but they won't make it very easy to debug a problem when getting an error report from a user. To make my life easier in such situations I add unique references whenever an error happens.

For that, I extend the original GeneralException class:

type ErrorReference = String

class GeneralException(message: String, cause: Throwable, errorReference: ErrorReference = GeneralException.createReference())
  extends Exception(s"$message | error-reference: $errorReference", cause)

object GeneralException {
  def apply(message: String, cause: Throwable = null, errorReference: ErrorReference = GeneralException.createReference()) = new GeneralException(message, cause, errorReference)
  def createReference(): ErrorReference = "ER_" + scala.util.Random.alphanumeric.take(12).mkString
}

Be aware that this code is not pure functional and is executes an effect at the time of the error-creation (generating a random reference). Again I believe that this is a good trade-off for the ease of use.
Depending on the system it is worth to spend some time to come up with an identifier format that e.g. includes information about time and hence is sortable and has fewer collisions... just in case there are a lot of errors - I hope there aren't :^).

Also note that the error-reference is automatically appended to the message, so it is guaranteed to appear in the stack trace when an error is logged.

I usually add this reference to my API error-types and then expose it to the end-user in case they want to report an error.

Conclusion

A few best practices are violated here, including class inheritance, usage of null and non-pure-functional code to create references on error-creation.
But overall, with only a little extra effort I get the best of both worlds: clean error-handling for both reading and usage, as well as comprehensive stack traces that make debugging easy.

Types to replace parameter names

2022-10-07T00:00:00Z

It's a classical mistake. You have defined a method that accepts some parameters and you forward them to another method.

public createUser(email: string, password: string) {
  return this.userRepository.create(email, password);
}

Is the order of arguments really correct? Or did we by accident mix up the user and the password?

After causing a bug by mixing up the parameter order, we eventually use the typesystem to our advantage to prevent this from happening again.
Instead of using stringly typed parameters, we define proper types:

public createUser(email: Email, password: Password) {
  return this.userRepository.create(email, password);
}

That's better. Now, if the order of parameters is different between our createUser method and the repository we use, we will get a compile error.
At least if we used actually classes and not just type aliases - but this is more of typescript-specific problem not a general one.

This is cool, but when applying this style broadly, we quickly end up with quite verbose code, repeating names for both identifiers and types all over the place.

One might argue that this is okay, because we might want to use the same type multiple times. For instance, if the user where to also provide an email-address for recovery, the method might look like this:

public createUser(email: Email, password: Password, recoveryEmail: Email) {
  return this.userRepository.create(email, password, recoveryEmail);
}

However, now we are back to our original problem and might mix up the regular email with the one for recovery purposes.
Consequently, we should make our types more precise again and have specific types for the email addresses, e.g.:

public createUser(email: DefaultEmail, password: Password, recoveryEmail: RecoveryEmail) {
  return this.userRepository.create(email, password, recoveryEmail);
}

But now we also have the same kind of verbose, duplicated code again.

Kill all the identifiers

Which brings up a question: if we meticulously follow the pattern of creating distinct types for specific entities, do we then still need identifiers at all?
That's an interesting thought-experiment. What if we just were to remove those parameter-identifiers from language specification alltogether?

Here is how that could look like:

public createUser(DefaultEmail, Password, RecoveryEmail) {
  return this.userRepository.create(DefaultEmail, Password, RecoveryEmail);
}

The parameter names are gone and to call the repository, we simply pass them by referring to the type-name instead.

Looks quite clean if you ask me. We also have the same type-safety guarantees as before.

But what about the code that we usually write within a method. We need to manipulate data, create temporary variables and so on.

Here's an example with the original code:

public createUser(email: DefaultEmail, password: Password, recoveryEmail: RecoveryEmail) {
  const normalizedEmail = email.toLowerCase();
  return this.userRepository.create(normalizedEmail, password, recoveryEmail);
}

We certainly need variable names here to disambiguate between the price and the discount. Imagine this to be production-code.
Maybe you wonder about the meaning of the number for the discount. Is that a fixed value as in $2 or is it a percentage or something else?

Since we are using a typesystem we should take advantage and improve our code with meaningful types:

public calculateTax(price: Price, discount: Discount, mode: OrderMode): Tax { ... }

It's not quite obvious what Price and Distount are exactly, but we can easily check how they are defined.
The type definitions might be defined like that:

type Price = Money
type Discount = Money
type Money = number
type OrderMode = "takeout" | "eatin"

Now looking at the code, it appears that we don't really need the variable names anymore.

public calculateTax(Price, Discount, OrderMode): Tax { ... }

Obviously that's illegal syntax in typescript, but let's go along with it for now.
The next question is how the implementation would look like.
Somehow we need to refer to the parameters. Here's how it could work:

public calculateTax(Price, Discount, OrderMode): Tax {
   const base = Price - Discount;
   if (OrderMode === "takeout") {
      return base * 0.1;
   } else if (OrderMode === "eatin") {
      return base * 0.2;
   }
}

The parameter-values are accessed just by their typenames.

Conflicts

This works in the example but what if we have a method where the types are just equal? For instance division:

public minus(a: number, b: number): number {
   return a - b;
}

Actually, if we try really hard to remember math classes back in the day, maybe we remember that there are already terms we can use:

public minus(Minuend, Subtrahend): Difference {
   return Minuend - Subtrahend;
}

But programming languages need to be practical. We don't want to define a new type for such a case everytime we encounter such a situation.
Even if it maybe be "correct" to choose a distinct name in those cases, it is often just not practical and will make our code bloated since we need to define the types in addition to the method.

So just like we can just create identifiers out of thin air, let's allow the same for types.

public minus(Minuend: number, Subtrahend: number): number {
   return Minuend - Subtrahend;
}

Now, that looks like we are back to parameter names.
But don't forget: we changed typescript the language and removed parameter names.
Instead, we have just created two new types inline, as if we had defined type Minuend = number before.

This means we can now mix predefined types and types that we need to create on the fly:

public calculateTax(Price, Discount, OrderMode: "takeout" | "eatin"): Tax { ... }

Price and Discount have been defined in advance, but maybe OrderMode wasn't. So we create it inline.

The call-side

Until now, there isn't really a big advance.
The method signature gets a little bit shorter and it nudges us to use distinct, semantic types instead of generic types but that's about it.

The real difference happens at the call-side.

First of all, since our types are unique, the ordering stops mattering.
We can freely reorder our parameters both at call-side and at definition side.

const price: Price = ...
const discount: Discount = ...

calculateTax(price, discount, "takeout") // that's fine

// Just as good
calculateTax(discount, price, "takeout")
calculateTax("takeout", price, discount)

If you feel that it is inconsequent to remove parameter names but still use variables then I can only say that you have a very good gut feeling indeed.
I'll cover this later on, but let's keep on with this style for a bit.

Of course, if we are in the context of another method, maybe price and discount are already parameters!

public calculateFinalPrice(Price, Discount, CustomerPreferences): Money {
  return (Price - Discount) + calculateTax(Price, Discount, CustomerPreferences.orderMode);

Wait a second - how would the compiler know, that the result can be the Money type?
And also, if we have a method defined with semantic types that are both numbers, how to turn numbers into those specific types at call-site?

There are two parts here. Let's look at the call-site first.

If want to call our method with two hardcoded numbers in code, how would that look like?

calculateTax(40, 15, "takeout")

This obviously won't do it.
In the current typescript world we could do something like that, but not in our new types-only world.
We have turn the values of type number into the correct types by converting them explicitly.

`calculateTax(40 as Price, 15 as Discount, "takeout" as OrderMode)`

We are reusing typescript's as keyword here. The compiler checks if Price is a specialized type of number and converts it.
This also gets us back to being able to reorder parameters as we wish without changing semantics.

It should be noted that this is only necessary when going from a general type like number to a more specialized like Price.
Going from Price to Money or from one of those to number would be automatic. Subtyping rules apply.

This might seem annoying at first, but such conversions become quite rare in most code if types are well utilized and reused.
It is a good incentive to keep the defined types tidy and organized and is incredible for self-documenting code (even though it can't replace all documentation obviously).

What about variables and return types?

So far we have made a thought-experiment mainly about changing method signatures. Why not extend this to variables as well? And also return types. I don't see why not.

Going back to a previous example, which we have to refine now:

const price: Price = ...
const discount: Discount = ...

calculateTax(price, discount, "takeout" as OrderMode)

We can also reduce this to:

const Price = 40
const Discount = 15

calculateTax(price, discount, "takeout" as OrderMode)

And we could reduce it even further:

const 40 as Price
const 15 as Discount
calculateTax(price, discount, "takeout" as OrderMode)

Class fields and names and method names

Why stop here?

Instead of defining a user as

class User {
  public firstName: string;
  public lastName: string;

  public constructor(firstName: string, lastName: string) {
    this.firstName = firstName;
    this.lastName = lastName;
  }
}

we write it like that in our new world:

class User {
  public firstName: FirstName;
  public lastName: LastName;

  public constructor(firstName: string, lastName: string) {
    this.firstName = firstName as FirstName;
    this.lastName = lastName as LastName;
  }
}

As you can probably see, this looks quite strange. And that is good! It is an indication of bad design.
Instead of doing the transformation ourselves, we should push the types out into the world!
In a class definition, the compiler would do that for us:

class User {
  // We just created two new public and refined types
  public FirstName: string;
  public LastName: string;

  public constructor(FirstName, LastName) {
    this.FirstName = FirstName;
    this.LastName = LastName;
  }
}

Now we create users and methods with those types:


const User("doe" as FirstName, "john" as LastName) as John;

public switchNames(User): User {
  return new User(User.FirstName as LastName, User.LastName as FirstName);
}

const switchName(john) as CorrectedJohn

Serialized Gedanken

Fundamental estimation problems

Estimations #

Single number estimations #

Estimations are no divinations #

Tackling the shortcomings #

Buffers #

Splitting tasks #

Side channels #

Confidentiality intervals #

How to use confidentiality intervals #

Bonus #

SQL as API

SQL in the API??? #

There is always an exception #

SQL to the rescue? #

Defining the data structure #

Processing the SQL #

Wrapping up #

Hold on! #

Documentation and error-handling #

Performance considerations #

Vendor lock-in #

Last words #

Scala's great ecosystem

Nothing ever work... mostly #

The challenge #

Graphql subscriptions to the rescue #

The happy surprise #

Context-dependant code is better than its reputation

Contexts outside of programming #

Contexts in programming #

Usage in practice #

Using ± to describe choices #

Using ± in another context #

A word about naming #

Conclusion #

Functional error-handling with stack traces

Why exceptions? #

Using it in code #

The resulting logs #

API errors #

Adding references and codes #

Error codes #

Error references #

Conclusion #

Types to replace parameter names

Kill all the identifiers #

Conflicts #

The call-side #

Type refinement and coercion #

What about variables and return types? #

Class fields and names and method names #

Estimations

Single number estimations

Estimations are no divinations

Tackling the shortcomings

Buffers

Splitting tasks

Side channels

Confidentiality intervals

How to use confidentiality intervals

Bonus

SQL in the API???

There is always an exception

SQL to the rescue?

Defining the data structure

Processing the SQL

Wrapping up

Hold on!

Documentation and error-handling

Performance considerations

Vendor lock-in

Last words

Nothing ever work... mostly

The challenge

Graphql subscriptions to the rescue

The happy surprise

Contexts outside of programming

Contexts in programming

Usage in practice

Using ± to describe choices

Using ± in another context

A word about naming

Conclusion

Why exceptions?

Using it in code

The resulting logs

API errors

Adding references and codes

Error codes

Error references

Conclusion

Kill all the identifiers

Conflicts

The call-side

Type refinement and coercion

What about variables and return types?

Class fields and names and method names