Of all the OTP abstractions, GenServer is the most iconic one. As a result, it is usually the first thing newcomers learn to use, and they inevitably run into some common pitfalls. Let's talk about one of them.

GenServers are commonly used whenever there is a need for a stateful process that will store some data and make it available to other processes. As an example, let's imagine we want to fetch currency conversion rates and make them available in our app. A naive approach could look something like this:

Mix.install([:req])

defmodule FXRates do
  use GenServer

  def start_link(arg, name \\ __MODULE__) do
    GenServer.start_link(__MODULE__, arg, name: name)
  end

  def get_rates(name_or_pid \\ __MODULE__) do
    GenServer.call(name_or_pid, :get_rates)
  end

  @impl GenServer
  def init(_init_arg) do
    {:ok, [], {:continue, :update_rates}}
  end

  @impl GenServer
  def handle_continue(:update_rates, _state) do
    %{status: 200, body: rates} = Req.get!(
      "https://api.frankfurter.dev/v1/latest"
    )

    Process.send_after(self(), :refresh_rates, to_timeout(minute: 1))
    {:noreply, rates}
  end

  @impl GenServer
  def handle_info(:refresh_rates, state) do
    {:noreply, state, {:continue, :update_rates}}
  end

  @impl GenServer
  def handle_call(:get_rates, _from, state) do
    {:reply, state, state}
  end
end

defmodule Main do
  def main do
    children = [
      FXRates
    ]

    {:ok, _} = Supervisor.start_link(children, strategy: :one_for_one)
  end
end

For convenience, let's put it into a single genserver.exs and play with it using iex -r genserver.exs. Indeed, we can start the supervisor and call FXRates.get_rates() to get the latest rates:

iex(1)> Main.main()
{:ok, #PID<0.249.0>}
iex(2)> FXRates.get_rates()
%{
  "amount" => 1.0,
  "base" => "EUR",
  ...
}

Now, if you submit this for code review to any experienced engineer, they will immediately point out that using GenServer.call for reading data is not good. GenServer is a natural serialization point and concurrent callers will have to form a line to get their FX rates. Not to mention that reads will block the GenServer and might mess with its refresh timer.

But how do we make reads concurrent? Well, we just need to put rates somewhere where they can be accessed concurrently. ETS will do. But the thing with raw ETS is that you need to manage the table's lifecycle. And it also comes with a certain style of developer experience that Erlang modules have. What if I told you there is a way to use ETS without using ETS?

┌─────────────┐
│┌───────────┐│
││    The    ││
││ GenServer ││
││  is [IN]  ││
││           ││
││ ●         ││
││           ││   ╭──────╮ ╭───────╮ ╭──────╮ ╭───────╮ ╭──────╮
││           ││   │ read │ │ write │ │ read │ │ write │ │ read │
││           ││   ╰──────╯ ╰───────╯ ╰──────╯ ╰───────╯ ╰──────╯
│└───────────┘│
└─────────────┘

Meet Registry

Registry is a way to register processes. But that's not all! It also allows those registrered processes to store arbitrary data. So, why don't we register our genserver in Registry and put FXRates in it?

Mix.install([:req])

defmodule FXRates do
  use GenServer

  def start_link(_arg, name \\ __MODULE__) do
    via_tuple = {:via, Registry, {FXRatesRegistry, name, %{rates: []}}}
    GenServer.start_link(__MODULE__, name, name: via_tuple)
  end

  def get_rates(name \\ __MODULE__) do
    [{_pid, %{rates: rates}}] = Registry.lookup(FXRatesRegistry, name)
    rates
  end

  @impl GenServer
  def init(name) do
    {:ok, %{name: name, rates: []}, {:continue, :update_rates}}
  end

  @impl GenServer
  def handle_continue(:update_rates, state) do
    %{status: 200, body: rates} = Req.get!(
      "https://api.frankfurter.dev/v1/latest"
    )

    Registry.update_value(FXRatesRegistry, state.name, fn _current ->
      %{rates: rates}
    end)

    Process.send_after(self(), :refresh_rates, to_timeout(minute: 1))
    {:noreply, %{state | rates: rates}}
  end

  @impl GenServer
  def handle_info(:refresh_rates, state) do
    {:noreply, state, {:continue, :update_rates}}
  end
end

defmodule Main do
  def main do
    children = [
      {Registry, keys: :unique, name: FXRatesRegistry},
      FXRates
    ]

    {:ok, _} = Supervisor.start_link(children, strategy: :one_for_one)
  end
end

There are a couple of changes here. First of all, we use a via tuple to register the GenServer. We also save name in the state: it acts as a Registry key, and we need to know it in order to update the data associated with our server. Apart from that, get_rates is not a GenServer call anymore! Try it in the IEx to confirm it still works.

Ok, but where are our rates stored now? In ETS, of course! Registry uses ETS under the hood, so the Registry.lookup call is just an ETS read. We made our reads concurrent without having to manage ETS table lifecycle ourselves.

Conclusion

Registry is an amazing tool and is a joy to work with. I reach for it every time I want to use a GenServer for serializing writes while keeping data available for concurrent reads. Give it a try!

Code reviews are dead and only now do we begin to notice. Hardly the only or most important casualty of the great AI leap, but an important one nonetheless.

For quite some time we had a good thing going and I will remember it. Maybe you remember it too or maybe you never experienced the tech industry before 2025 and want to know what it was like. Let me tell you how it was.

The Author

Code review is many things. On its mechanical, bodily level, it is an act of reading through a diff. A pull request. A unit of change that is asking for permission to enter the repository. It is a shared space and it must be kept in order. So you need to show respect.

A pull request should be presentable. You have to think about it long before you push your branch. It can't be too big – nobody likes that. It should have a description – this is where you provide additional context and explain anything that can't be easily inferred from the code. This is also the final destination for future code archeologists who will smell your blood in git blame and judge if your decisions stood the test of time.

So make it nice. Some pre-emptive comments are always appreciated, although they might be used against you as evidence of code not being clear enough. "I see you felt the need to clarify this piece of code, interesting," some reviewers might say.

Usually, there is also CI. A set of automated checks that run against your PR. Compile, lint, test. The first line of defense. You'd better make sure the build is green before assigning reviewers or you'll have to come up with sorry excuses. "Huh, CI is failing, I'll look into it later." Sloppy job and an easy way to part ways with some street cred. Unless the tests are flaky, of course, in which case you have a choice to rise to the occasion and fix them, or meld into collective inaction. Some people might remember what you chose.

When all rituals are complete, it's time to assign reviewers. This, too, can be delegated to a machine! Can be a single very important person, somebody with authority. You can bet they are too busy these days to write code themselves. Alternatively, a reviewer can be selected at random. Roll a die and let fate decide. Might even add some round robin rules on top, to spread the burden more evenly. What a terrible way to pick a reviewer!

My personal favorite is selecting reviewers myself. Some people assign the whole team! Rookie mistake. A sure way to never get a review. No, you need to pick a single person. One victim. Once you lock eyes, they will feel the burden and the honor in its entirety. No escape from the duel.

The Reviewer

For the reviewer, the diff is a net loss. A nuisance. A necessary, but fundamentally secondary activity. Congratulations, you're on gatekeeping duty! Time to put off your actual work and load a bunch of completely new context into your head. What torture! I just hope there is at least a description.

To be a reviewer is to be tempted. "Cut corners, skim through, LGTM" – you hear a voice inside. Hm... No, no, you can't. It's your job. And there might be consequences! If the code is bad, it poisons everything around it. One broken window will lead to another. Toxins accumulate in the body. You need to be strong. And that would be disrespectful, wouldn't it? The author put in the effort, after all. Let's see.

A good reviewer starts from the top. What is this change for? What is it trying to achieve? Is this the right thing? If so, is this the right approach? First comments can land before a single line of code is even read. Those are brutal, but most valuable.

Then, if everything looks good, time to look at the code. Don't look too closely though, we're not quite there yet. Just the big picture. This is what it is now, but what could it be? Maybe there's a better way to compose functions? A simpler model? How would you do this? A costly mental exercise, but a good author will appreciate it.

Finally, the trivial part. Just read the code, poke holes in the solution. This is a mixed bag of cute little TILs and nits, variable names and code golf. A token effort, but most common.

The Ritual

Code review is many things. It is a form of asynchronous communication. An avenue for knowledge sharing. A learning opportunity. A safety net.

It is also a battleground. And an asymmetric one! One side is defending, another attacking. Things can get heated. You need to be careful with that. Wrap comments in questions. Assume good faith and make self-deprecating jokes. Attach GIFs. Use objective measures when possible – it's often hard to agree on what is "simpler" or "more declarative". And never invoke your authority directly. That's in very bad taste.

At times, code reviews were proving grounds for people who just joined the team and are going through the inevitable period of friction. A "storming" phase, if you will. Code reviews turn into a ritual dance of testing the new guy and explaining to them our unspoken rules. A fresh pair of eyes is often quick to spot weaknesses, but you'd be foolish to point at them right away. A newcomer needs to be respectful and first demonstrate that they can learn existing ways and contribute on par with the rest of the team. Only then can you voice your concerns and propose changes. A lot of teams are aware of this, of course, and do their best to hold an egalitarian stance and not be defensive. But there's only so much success you can have against a primal force like this, so it's wise to always respect those who came before you.

For me, personally, code reviews were a place of exchange. As an author, I bring the product of my labor for other people to witness, hoping that all the effort and love I put into it will be seen and appreciated. As a reviewer, I am looking for the signs of this effort and in return, pay my respects to the author through thoughtful comments. Tomorrow, we will switch roles and the exchange will repeat.

No going back

Nothing about code reviews survived the AI transition. The code is not the product of human labor anymore, not in the same sense at least. The diffs aren't read with the same scrutiny. Some even argue the code isn't for human eyes at all, and thus readability doesn't matter. No point in sophisticated arguments and thoughtful comments: a coding agent will agree with anything you say and will not learn a thing. And even if you choose to code by hand, you will likely be met by an AI reviewer.

I know there is no going back. Even if some teams still review code the old ways, those are merely uncontacted tribes of the AI age. LLMs invaded the spaces previously inhabited exclusively by humans. They are omnipresent in every PR, every comment and every diagram. Even a team of devoted tradcoders won't feel the same, because they know what they are running from.

I don't know what will replace code reviews or how they will evolve, but I feel like we lost something valuable. I know I have. And I'm mourning.

There is a very popular opinion that coding agents just make senior devs more productive. AI is just a tool that allows you to off-load the boring part of writing code to the agent so you can focus on real engineering™. Nothing dramatic is happening, just another addition to the toolbox.

And it's not like this isn't true. It's absolutely the case that great devs are more productive with AI than ever. But in the long run this is irrelevant, because developers as we know them are about to become a historical curiosity.

How do software engineers become "great"? Apparently, through a combination of learning, working, tinkering, open sourcing, talking, and whatnot. All the normal things we did before 2023, you know what I'm talking about. That was the journey of every single great engineer of past and present. But this won't be the journey of the future, because AI is already inserted in every single step of this pipeline.

The world where you were forced to learn before you can do things quite literally does not exist anymore. And I don't care if you still have what it takes to read a book from start to finish, we're talking about incentive structure change for our group as a whole. Sure enough, there will be new things to learn and a new journey to take. Maybe it will be better or worse, that's irrelevant for now. What's important is that it will be different and it will produce a different kind of engineers.

We all know about valuable pre-2022 datasets, but there's also a very limited stock of pre-2022 people. This means that the current situation where we have thousands of competent engineers "just using AI to be more productive" is a historical anomaly. We look at our own experience and extrapolate it, but we're already a thing of the past. It doesn't matter how we use AI.

The current state of AI affairs simply can't continue past the current generation. Something will have to change. Maybe we'll figure out a new way to learn that is compatible with AI. Maybe AI will become so good that humanity won't need people who understand how things actually work. Or maybe we'll move universities off the grid Anathem style. I don't know what it will be, but it will definitely not be what we see now.

And yes, I know about fundamental technological advancements of the past. I don't care how similar AI is to printing press, or steam engine, or photography, or internet, it's also very different. AI is a genuinely novel technology and our generation has a duty to figure it out. It might take updating our etiquette or developing new tech, but we need to take this seriously.

The PostHog Elixir SDK 2.0 just dropped.

What is PostHog? Why should you care about its SDK? Why do I think it's good? Bear with me, I'm about to tell you how I spent my summer.

Dear Diary

In May 2025 I found myself with a lot of free time and an appetite for writing a logging library from scratch. How exactly I developed this appetite is a whole other story, but the bottom line is that I wasn't happy with how all existing logging libraries shared a lot of design decisions and workarounds, a lot of which were based on solvable problems or not relevant anymore.

And right when I was entertaining these thoughts, PostHog released its Error Tracking solution to open beta. This was a perfect opportunity to write a library.

So I bought a Claude Code subscription and one-shotted the whole thing in two days. Actually, no. I locked in and put a ton of effort into it. I learned everything I could about Elixir logging. Whenever I hit a roadblock, I would take a step back and fix it at the origin, whether it was in Plug, LoggerJSON or Elixir itself. I also documented everything and released it as an educational Hex package LoggerHandlerKit.

All of this resulted in LogHog – an okay library focused solely on PostHog Error Tracking.

It didn't take long for PostHog's very own Elixir champion Rafa Audibert to reach out. Naturally, error tracking would be a good addition to the official library. Things quickly got out of control though, and a couple of months later we have a new, completely overhauled version of the SDK on our hands. And, dear reader, I think it's good.

What to expect

So what is PostHog and why does it need an SDK? After all, the Elixir ecosystem is not big on SDKs. Just using Req is a great experience and sets a pretty high bar to clear for any bespoke library.

I'm glad you asked! Let me walk you through what PostHog brings to the table for an Elixir application.

Event Capture

I don't work for PostHog, so I'm going to tell you the truth. PostHog is a big ClickHouse cluster run by a team whose life goal is to see how many products they can build on top of it.

The most important, core PostHog feature is the ability to send events to the server, where they will be accessible through an SQL-like query language. Each event can have properties attached to it. You can shove anything in there. The only required property is distinct_id, which identifies users, persons, or whatever you decide it does. Events happen to someone, after all.

Just emitting events is already great and is what we would call product analytics. But PostHog has made events even more useful! Some events have special meaning and experiences built for them. If you emit $exception_list events, they will be available in Error Tracking. Capture $ai_generation and you have LLM Analytics.

Events are very versatile, and PostHog is building more and more products every day.

Context

Sending analytics events is not as simple as sprinkling HTTP requests over your codebase. You want to instrument your events with properties and they originate in very different places. In order to note a user's IP address, you need to tap into your Endpoint. For tenant ID, look inside the authentication stack.

If we had to pipe all relevant information through all the layers of our app, we would go insane. This is why there is an established pattern in Elixir to use the process dictionary for that kind of metadata.

The PostHog SDK comes with a context mechanism to address exactly this. You can set PostHog context and it will be attached to events captured in the same process.

PostHog.set_context(%{distinct_id: "distinct_id_of_the_user"})
PostHog.capture("page_opened")

Batching

Capturing events is important, but usually not important enough to block your app code until the API request returns. That's why you want to offload sending them to some background task. Sure, Task.start is always there, but it would also be nice to batch events together...

PostHog SDK manages a pool of sender processes for you that automatically batch and send all the events you capture. So when you call PostHog.capture, it's merely a GenServer.cast under the hood.

Integrations

Some things are solved problems. PostHog gives special meaning to properties like $current_url or $host, and the best place to grab those is from the Plug.Conn struct. This knowledge is already baked into the SDK as PostHog.Integrations.Plug. All you need to do is add it to your pipeline.

For now this is the only integration, but it won't be long until there are more.

Product-specific API

Sometimes you just need a little helper. The feature flags product, for example, goes beyond simple event capture. The recommended flow to evaluate a feature flag is the following:

1. Make a call to /flags endpoint
2. Capture $feature_flag_called event
3. Use $feature/<flag_name> property for all events captured for this user.

The SDK encapsulates all of this knowledge and exposes it as a single PostHog.FeatureFlags.check function.

Error Tracking

Finally, the part that was transferred from LogHog. PostHog comes with a logger handler that captures relevant log events and makes them accessible in PostHog Error Tracking.

I have to say, though, that unlike other PostHog products, Error Tracking requires language-specific backend support for the best experience, and Elixir support is not yet implemented. But it works, so give it a try!

Test Infrastructure

You want to test your analytic events, right? Right?

Well, you're in luck because the PostHog SDK does the right thing and uses NimbleOwnership to scope all captured events to their owner processes. Expect your async tests to remain async:

test "capture event" do
  PostHog.capture("event", %{distinct_id: "distinct_id"})

  assert [event] = PostHog.Test.all_captured()

  assert %{
            event: "event",
            distinct_id: "distinct_id",
            properties: %{},
            timestamp: _
          } = event
end

Conclusion

The new PostHog SDK is here and I think it's good. It was written by people who care about both Elixir and PostHog. It's fresh off the press and not battle-tested yet, so tread with caution, but if you're looking to capture events in your app, maybe give it a try.

Caller tracking has been part of Elixir since 2018, and yet it remains a relatively obscure mechanism. Let's fix this with a 5-minute guide.

What is caller tracking

Asynchronous tests are the best. They are possible because in many cases we can trace what is happening in the application all the way back to the test. We say that a test process "owns" things, and other processes may be "allowed" to access those things.

Caller tracking is a convention that allows us to know which processes are related to each other and automatically allow them to access things owned by a test. Examples of such things are Ecto.Sandbox transactions and Mox mocks.

In some cases, caller tracking is handled automatically and you don't need to do anything, such as if you use the Task module. But in a lot of other cases you need to take care of caller tracking yourself. The most common example is a GenServer.

Genserver with caller tracking

Whenever you start a GenServer, equip it with a :"$callers" value in the process dictionary. It should include whatever the calling process had in its process dictionary under the same key and the caller PID. Remember: start_link is executed in the caller, while init runs in the server!

defmodule MyGenserver do
  use GenServer
  
  def start_link(args) do
    callers = Process.get(:"$callers") || []
    new_callers = [self() | callers]
    GenServer.start_link(__MODULE__, {new_callers, args})
  end

  @impl GenServer
  def init({callers, _args}) do
    Process.put(:"$callers", callers)
    {:ok, nil}
  end
end

Now whenever you start this GenServer in your test, it will be able to access Ecto transactions without any additional setup.

Further reading

Ancestor and Caller Tracking bit in Task module documentation

What Does NimbleOwnership Do Anyway?

How to Async Tests in Elixir

Gist with full single file example

Imagine you're a scrambler.

Imagine that you encounter a signal. It is structured, and dense with information. It meets all the criteria of an intelligent transmission. Evolution and experience offer a variety of paths to follow, branch-points in the flowcharts that handle such input. Sometimes these signals come from conspecifics who have useful information to share, whose lives you'll defend according to the rules of kin selection. Sometimes they come from competitors or predators or other inimical entities that must be avoided or destroyed; in those cases, the information may prove of significant tactical value. Some signals may even arise from entities which, while not kin, can still serve as allies or symbionts in mutually beneficial pursuits. You can derive appropriate responses for any of these eventualities, and many others.

You decode the signals, and stumble:

I had a great time. I really enjoyed him. Even if he cost twice as much as any other hooker in the dome—

To fully appreciate Kesey's Quartet—

They hate us for our freedom—

Pay attention, now—

Understand.

There are no meaningful translations for these terms. They are needlessly recursive. They contain no usable intelligence, yet they are structured intelligently; there is no chance they could have arisen by chance.

The only explanation is that something has coded nonsense in a way that poses as a useful message; only after wasting time and effort does the deception becomes apparent. The signal functions to consume the resources of a recipient for zero payoff and reduced fitness. The signal is a virus.

Viruses do not arise from kin, symbionts, or other allies.

The signal is an attack.

– Peter Watts, "Blindsight"

In the sci-fi novel Blindsight by Peter Watts, humanity encounters scramblers – an alien species that is intelligent, but lacks consciousness. Scramblers do not attempt to contact humans and wage a unilateral total war against us. Why? Because we talk too much. Our little planet just can't stop chatting about nonsense, and scramblers know only one use for information: to perceive it. They can't help but listen to all our yapping that only wastes their precious brain cycles and reduces their chances of survival. No peaceful species would do this; this is clearly an act of war.

When I first read Blindsight 10 years ago, I didn't understand how scramblers felt about human slop. But now I do. I feel it every time I read AI text outside of the privacy of my own chatroom.

Proof-of-thought

For the longest time, writing was more expensive than reading. If you encountered a body of written text, you could be sure that at the very least, a human spent some time writing it down. The text used to have an innate proof-of-thought, a basic token of humanity.

Now, AI has made text very, very, very cheap. Not only text, in fact. Code, images, video. All kinds of media. We can't rely on proof-of-thought anymore. Any text can be AI slop. If you read it, you're injured in this war. You engaged and replied – you're as good as dead. The dead internet is not just dead it's poisoned. So what do we do?

Luckily for us, AI only talks in response. Unlike Earth, AI does not emit comedy sketches into outer space on its own. To get AI slop, somebody needs to ask for it. To send it further, someone needs to retransmit it. Our problem is other humans, really.

There's nothing wrong with using AI. When you do, you know what you're getting. The transaction is fully consensual. But whenever you propagate AI output, you're at risk of intentionally or unintentionally legitimizing it with your good name, providing it with a fake proof-of-thought. In some cases, it's fine, because you did think it through and adopted the AI output as your own. But in other cases, it is not, and our scrambler brain feels violated.

AI etiquette

╭─────────────────────────╮
│ I asked ChatGPT and     │
│ here's what it told me: │
│ ...                     │
╰─────────────────────────╯
                                    ╭─────────╮
                                    │ blocked │
                                    ╰─────────╯

I think that realistically, our main weapon in this war is AI etiquette. My own take on AI etiquette is that AI output can only be relayed if it's either adopted as your own or there is explicit consent from the receiving party. There's plenty of discussion to be had about what adopting entails and what can be considered consent, but I believe the core principle is sound.

"I asked ChatGPT and this is what it said: <...>".

Whoa, let me stop you right here buddy, what you're doing here is extremely, horribly rude.

"I had a helpful chat with ChatGPT about this topic some time ago and can share a log with you if you want."

Yeah, send it my way I'll take a look.

"I vibe-coded this pull request in just 15 minutes. Please review"

Well, why don't you review it first?

"Here's my PR, I did this and that for this and that reason."

Thank you, I'll take a look.

Unlike scramblers, we're only exposed to meaningless noise by choice. Be polite, and don't send humans AI text.

06:04:59.473 [error] ** :"myapp@myhost": Connection attempt from node :AAAAAA@AAAAAAA rejected since it cannot handle [~c"UNLINK_ID", ~c"HANDSHAKE_23", ~c"BIG_CREATION", ~c"BIT_BINARIES", ~c"EXPORT_PTR_TAG"].**

This is NOT what you want to suddenly pop up in your logs at 10 PM.

I am a simple man. I host my silly little hobby projects on a single VPS server behind an NGINX proxy. There is nothing sensitive there, no user data; they don't even user cookies! I'm sure I'll be fine with the perfectly sensible security defaults, right?

Well, yes and no. Yes, nothing bad happened so far. No, I'm not totally fine; this log message almost gave me a heart attack.

        ┌───┬───────┬───────┬───────┬───────┬───┐
        │   │       │       │       │       │   |
        ├───┴───┬───┴───┬───┴───┬───┴───┬───┴───┤
        │       │       │       │       │       |
        ├───┬───┴───┬───┴───┬───┴───┬───┴───┬───┤
        │   │   ┌───┴───────┴───────┴───┐   │   |
        ├───┴───┤       WANTED          ├───┴───┤
        │       ├───────────────────────┤       │
        ├───┬───┤                       ├───┬───┤
        │   │   │                       │   │   │
        ├───┴───┤                       ├───┴───┤
        │       │                       │       │
        ├───┬───┤   AAAAAA@AAAAAAA      ├───┬───┤
        │   │   │                       │   │   │
        ├───┴───┤                       ├───┴───┤
        │       │                       │       │
        ├───┬───┤                       ├───┬───┤
        │   │   ├───────────────────────┤   │   │
        ├───┴───┤ FOR UNAUTHORIZED      ├───┴───┤
        │       │ ACCESS ATTEMPT        │       │
        ├───┬───┴───┬───────┬───────┬───┴───┬───┤
        │   │       │       │       │       │   │
        ├───┴───┬───┴───┬───┴───┬───┴───┬───┴───┤
        │       │       │       │       │       │
        └───────┴───────┴───────┴───────┴───────┘

Who is :AAAAAA@AAAAAAA?

Ah yes, this is just the right amount of As to go and search for it on the internet. It isn't hard, so I'll spare you the details. Apparently there was a CouchDB exploit that allowed for an attacker to perform a remote code execution by connecting to an Erlang node. It used AAAAAA@AAAAAAA as the fake node name. And when there is an exploit, there is automation. Somebody out there running this exploit (and no doubt many others) looking for unsecured Erlang nodes to connect to. My app node rejected the connection, but the attempt got close enough to be logged and we definitely don't want that.

I am the AAAAAA@AAAAAAA now

Before we fix anything, we need to reproduce the problem. Theoretically, we can just run the Python script from the link above, but that's no fun. Let's go step by step. This will also help us implement security measures later.

First of all, epmd. The very first thing the hacker is doing is looking if there's even an Erlang node on the server. For this, they just ask epmd through the default port (4369). We can do this with netcat:

echo -ne "\x00\x01\x6e" | nc <IP> 4369
name my_app at port 38255
name another_app at port 43493

Huh, you guys aren't hiding from the world at all, are you? Anyway, now we have a list of nodes and their ports. Let's connect using the binary message from the script:

echo -ne "\x00\x15n\x00\x07\x00\x03\x49\x9cAAAAAA@AAAAAAA" | nc <IP> 38255

snot_allowed

Boom, we got it! The log message is the same, but we're the attacker now!

Hobby-grade Security Plan

Time to devise and implement our hobby-grade security plan. And I say hobby-grade for a reason. If you're truly running a distributed cluster in Kubernetes or what not, you won't find any advice here.

1. Disable Erlang Distribution?

If we're running a single node, we don't need Erlang distribution, right? Welllll. Strictly speaking, we don't, but it's required for some nice bin/myapp commands, such as remote. I personally can't give up on that, but if you can, look for the RELEASE_DISTRIBUTION environment variable in the release.

2. Disable epmd

OK, we might want to run a distribution, but can we at least not advertise our nodes to the world with epmd? Yes! The keyword here is "epmdless". Basically, you can disable automatic epmd start and hardcode a port your app will be using.

But don't be fooled, even though epmd isn't there to snitch, the port is still open and discoverable:

nmap -sV -Pn -n -T4 -p- <IP>
Starting Nmap 7.95 ( https://nmap.org ) at 2025-03-01 13:01 PST
Nmap scan report for <IP>
Host is up (0.028s latency).
Not shown: 65529 closed tcp ports (conn-refused)
PORT     STATE SERVICE        VERSION
...
6789/tcp open  ibm-db2-admin?

Nooooo, this means attackers can still attempt to connect to the node! It just got harder to do!

3. Firewall

I regret to inform you, you might actually have to configure a firewall. At least that's what I did! I can't help you much here, except for maybe a little glimpse of hope: it wasn't actually that hard to do.

Was There Ever a Threat?

Kind of. There are a couple of hoops the attacker would need to jump through in order to actually connect, like dealing with short node names and knowing the magic cookie, but Erlang documentation states that those mechanisms aren't designed to ensure security.

Conclusion

There is someone out there dressed up as an Erlang node going from server to server knocking on port 4369. You're probably safe, but the knock is creepy nonetheless. Make sure you keep your epmd under control and your ports closed, otherwise... :AAAAAA@AAAAAAA WILL GET YOU!

If you've worked with Elixir for a while, you know that there are two kinds of tests: the good ones and the ones that need to be justified. The former being, of course, asynchronous tests – fast and joyful, and the latter being synchronous tests, the ones that get a side eye in code reviews.

Anyone who tries to write asynchronous tests sooner or later learns about the concept of ownership. This concept exists in Ecto, Mox, Req, and other libraries. In fact, it became so common that ownership management was extracted into a separate library called NimbleOwnership.

In this post, we'll see how the idea of ownership may come up naturally and what part of it can be handled by NimbleOwnership.

Hell Is Other Tests

When we say "ownership" we mean "resource ownership". You see, asynchronous tests exist in a highly concurrent environment, where anything can happen. Other tests might write to the same database (Ecto) or call the same mocks (Mox, Req) as your target test. When this happens, it may really mess up your test expectations.

To demonstrate this, let's look at another type of global resource: handlers - specifically logger handlers. Let's say we have a logger handler that simply counts the number of logs with an Agent process:

ExUnit.start()

defmodule OtherTests do
  use ExUnit.Case, async: true
  require Logger

  for i <- 1..10 do
    test "test #{i}" do
      Process.sleep(100)
      Logger.info("Other tests")
    end
  end
end

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  @behaviour :logger_handler
  @impl :logger_handler
  def log(_event, %{config: agent}) do
    Agent.update(agent, fn counter -> counter + 1 end)
  end

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    :logger.add_handler(:handler_id, __MODULE__, %{config: agent})
    Process.sleep(100)
    Logger.info("Foo")
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

With the power of Process.sleep/1, we can carefully craft this concurrency train wreck. The test in the MyTest module attaches a logger handler that is called whenever we log something. It then logs a single message and asserts that the counter was, in fact, incremented. Unfortunately, there are other asynchronous tests that aren't shy about logging messages themselves, and those messages also call the same logger handler, which interferes with our test:

1) test test log (MyTest)
   ex_unit.exs:24
   Assertion with == failed
   code:  assert Agent.get(agent, fn counter -> counter end) == 1
   left:  2
   right: 1
   stacktrace:
     ex_unit.exs:29: (test)

We can't let this happen, can we?

Can I See Your PID?

Ok, all we need to do is not let other tests call our logger handler. Luckily for us, handlers are executed in the same process, so we can just add the test PID to the handler config and check it:

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger
  
  @behaviour :logger_handler
  @impl :logger_handler
  def log(event, %{config: {agent, test_pid}}) do
    with true <- test_pid == self() do
      Agent.update(agent, fn counter -> counter + 1 end)
    end
  end
  
  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    :logger.add_handler(:handler_id, __MODULE__, %{
      config: {agent, self()}
    })
    Process.sleep(100)
    Logger.info("Foo")
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

Problem solved! No, seriously, problem solved. OK, OK, I hear what you're saying - in real life, logger handler code is in your application and adding a PID check to it is silly. That's fair. But we can always create a wrapper!

defmodule MyApp.LogHandler do
  @behaviour :logger_handler

  @impl :logger_handler
  def log(_event, %{config: agent}) do
    Agent.update(agent, fn counter -> counter + 1 end)
  end
end

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  @behaviour :logger_handler
  @impl :logger_handler
  def log(event, %{config: {{real_handler, og_cfg}, test_pid}} = cfg) do
    with true <- test_pid == self() do
      real_handler.log(event, Map.put(cfg, :config, og_cfg))
    end
  end

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    :logger.add_handler(:handler_id, __MODULE__, %{
      config: {{MyApp.LogHandler, agent}, self()}
    })
    Process.sleep(100)
    Logger.info("Foo")
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

or even better, a logger filter:

defmodule MyApp.LogHandler do
  @behaviour :logger_handler

  @impl :logger_handler
  def log(_event, %{config: agent}) do
    Agent.update(agent, fn counter -> counter + 1 end)
  end
end

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)

    :logger.add_handler(:handler_id, MyApp.LogHandler, %{
      config: agent,
      filters: [
        ownership_filter: {
          fn event, test_pid ->
            if test_pid == self(), do: event, else: :stop
          end,
          self()
        }
      ]
    })

    Process.sleep(100)
    Logger.info("Foo")
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

It works, but at this point you probably see the limitation. Not all processes are our enemies. Some of them are alright, in fact. Really chill processes. But our solution is relentless:

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    test_pid = self()

    :logger.add_handler(:handler_id, MyApp.LogHandler, %{
      config: agent,
      filters: [
        ownership_filter: {
          fn event, test_pid ->
            if test_pid == self(), do: event, else: :stop
          end,
          test_pid
        }
      ]
    })

    Process.sleep(100)
    task = Task.async(fn -> Logger.info("Foo") end)
    Task.await(task)
    
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

1) test test log (MyTest)
   ex_unit_spawn.exs:28
   Assertion with == failed
   code:  assert Agent.get(agent, fn counter -> counter end) == 1
   left:  0
   right: 1
   stacktrace:
     ex_unit_spawn.exs:53: (test)

╭─────────────────────╮
│ Don't talk to me or │
│ my son ever again   │
╰─────────────────────╯
                   │  ╭────────────────╮
                   ╰─ │                │
                      │ Logger Filter  │╭────────────────╮
                      │                ││ Logger Handler │
                      ╰────────────────╯╰────────────────╯

Sharing Is Caring

How do we allow other processes to access our beloved resources? Let's think about it – we first create a filter that knows to only proceed if it is called in a process with a specific pre-defined PID. So, if we want to allow another PID, we need to expand this from a single PID to a list of allowed PIDs. The problem is that spawned processes' PIDs aren't known at the time when we define the filter. Well, in that case, we can tell the filter to check not the PID itself, but some sort of secret. A token of process friendship that we'll put into any spawned process's dictionary.

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    test_pid = self()

    :logger.add_handler(:handler_id, MyApp.LogHandler, %{
      config: agent,
      filters: [
        ownership_filter: {
          fn event, _config ->
            if Process.get(:friendship_token), do: event, else: :stop
          end,
          nil
        }
      ]
    })

    Process.sleep(100)

    task =
      Task.async(fn ->
        Process.put(:friendship_token, true)
        Logger.info("Foo")
      end)

    Task.await(task)
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

See, it works! All we need to do is hand out this token to all processes that are relevant to the test and make sure the filter knows where to look for it.

Do You Know Who My Dad Is?

Not going to lie, this does sound like a chore. But have no fear, as Elixir has us covered! There is a convention that processes that are part of other processes' hierarchy in a meaningful way keep track of their ancestor PIDs in their dictionary. This list can be found under a magical key $callers. This is called caller tracking, and a lot of modules do this automatically. For example, the Task module or start_supervised/2 ExUnit helper. We can use this to our advantage: instead of having a custom token, we'll ask our filter to look for the test PID in the $callers list.

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    test_pid = self()

    :logger.add_handler(:handler_id, MyApp.LogHandler, %{
      config: agent,
      filters: [
        ownership_filter: {
          fn event, test_pid ->
            callers = Process.get(:"$callers") || []
            if test_pid in [self() | callers], do: event, else: :stop
          end,
          test_pid
        }
      ]
    })

    Process.sleep(100)
    task = Task.async(fn -> Logger.info("Foo") end)
    Task.await(task)
    
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

Since we know that Task.async/1 will add the test PID to the spawned process callers list, we can look for it in the filter. $callers is basically a name-dropping list. If you want to be at the party, you've got to tell them that you know people.

Facebook For Processes

So what does NimbleOwnership do anyway? Oh, it's just a Facebook for processes! It keeps track of friends lists in a large book. Here's a quick primer on what NimbleOwnership lets us do:

1. We can start a server, which is like a big book.
2. Processes can register arbitrary keys that symbolize resources. They are like different lists in the book, with every list being tied to a process that registered it.
3. Owner processes can allow other processes to access those keys. It's like adding a name to a list of invited guests. But don't be fooled, "allowance" here is just a name on the list. The server does not enforce this access in any way. It is a bookkeeper, not a bouncer.
4. Anyone can check if a process is on any list for a resource. This part is important and a little counterintuitive; we'll take a closer look at it after an example.

So, let's use NimbleOwnership for our test:

Mix.install([
  {:nimble_ownership, "~> 1.0"}
])

{:ok, _} = NimbleOwnership.start_link(name: OwnershipServer)
ExUnit.start()

defmodule OtherTests do
  use ExUnit.Case, async: true
  require Logger

  for i <- 1..10 do
    test "test #{i}" do
      Process.sleep(100)
      Logger.info("Other tests")
    end
  end
end

defmodule MyApp.LogHandler do
  @behaviour :logger_handler

  @impl :logger_handler
  def log(_event, %{config: agent}) do
    Agent.update(agent, fn counter -> counter + 1 end)
  end
end

defmodule MyTest do
  use ExUnit.Case, async: true
  require Logger

  test "test log" do
    {:ok, agent} = Agent.start_link(fn -> 0 end)
    
    NimbleOwnership.get_and_update(
      OwnershipServer, self(), :handler_id, fn nil ->
      {nil, %{}}
    end)

    :logger.add_handler(:handler_id, MyApp.LogHandler, %{
      config: agent,
      filters: [
        ownership_filter: {
          fn event, handler_id ->
            callers = Process.get(:"$callers") || []
            
            OwnershipServer
            |> NimbleOwnership.fetch_owner([self() | callers], handler_id)
            |> case do
              {:ok, _owner_id} -> event
              _ -> :stop
            end
          end,
          :handler_id
        }
      ]
    })

    Process.sleep(100)    
    task = Task.async(fn -> Logger.info("Foo") end)
    Task.await(task)
    
    assert Agent.get(agent, fn counter -> counter end) == 1
  end
end

There aren't many changes. Let's break them down.

First, at the very beginning, we start the OwnershipServer process - just a big book.

Then, at the beginning of the test, before we do anything of interest, we create a key :handler_id in the server. This key is the resource we're talking about. A list in the book with just one name on it – the test PID, which created the list.

Our logger filter is both the bouncer and a partygoer. It finds itself inside a task and all it knows is that it wants to knock on the door with a :handler_id label on it, but only if it is invited. So it goes to the OwnershipServer (everybody knows OwnershipServer, a nice chap and very popular) and asks a question you won't normally hear at a party: "Hey, has anyone allowed me in? I know $callers by the way." And the answer is "Yes," because, even though our test did not allow the task explicitly, it allowed itself. Or rather, it owns the place. So yeah, if you can name-drop the owner of the house, go ahead.

Note that, in the general case, processes can create lists for the same key. So, when we ask the server if anyone allowed us in, we don't really care who allowed us. All we care about is that somebody, somewhere is expecting us. 🥰🥰🥰

For this reason, keys are often unique to the test, and that's what we rely on in our example. If some other test creates a :handler_id resource, our filter will allow it too. In some cases, this is useful, but for our handler example, we wouldn't want that to happen.

Conclusion

Most of the time, you won't have to deal with low-level ownership machinery. But understanding what caller tracking is and how it propagates is vital to writing asynchronous tests. That said, if you ever find yourself testing handlers or other global resources, remember that it's very doable to limit access to them even in a highly concurrent environment. You'll just need a good bouncer.

Dear reader, I am easily confused by loggers. For a long time, the Elixir logger was simple enough that I could live with it. But starting with version 1.15, something changed, and now I'm confused.

Luckily for me, Elixir.Logger documentation does an excellent job explaining everything. The problem is that it's a lengthy read and I need a refresher the moment I take my eyes off it.

This post is an attempt to complement the official docs by not adding more documentation, but rather by taking things out.

About Logger Handlers

• Elixir Logging is configured using logger handlers.
• A logger handler is an Erlang concept.
• The default logger handler is :logger_std_h
• Logger handlers are called in the client process.
• It is recommended that logger handlers implement something called overload protection.
• :logger_std_h does implement overload protection.
• Logger handlers are configured under your application's namespace:

  config :my_app, :logger ...
  

• The default logger handler can be configured under the :logger namespace:

  config :logger, :default_handler ...
  

• Non-default logger handlers need to be attached to your application on startup

  Logger.add_handlers(:my_app)
  

About Logger Backends

• LoggerBackends is a Hex package.
• LoggerBackends code is also included in core Elixir for backward compatibility.
• LoggerBackends was the primary logging mechanism in Elixir before version 1.15.
• LoggerBackends provides a logger handler called LoggerBackends.Handler.
• LoggerBackends.Handler implements overload protection.
• A logger backend is a concept unique to LoggerBackends.
• A logger backend is a module that can be plugged into LoggerBackends.Handler.
• All logger backends benefit from overload protection implemented in LoggerBackends.Handler.
• Logger backends run in a single separate process.
• It is possible to unknowingly use LoggerBackends via legacy configuration options.

About Extension Points

• It is very rare that you need to implement your own logger handler.
• Logger handlers can be extended using Filters and Formatters.
• It is normal to employ multiple logger handlers.
• An example of a custom logger handler use case is Sentry.LoggerHandler.
• An example of a custom logger formatter use case is the LoggerJSON suite.

┌──────┐
│ Logs │
└──┬───┘
   │
   ├───────────────────┬───────────────────┐
   │                   │                   │
   ▼                   ▼                   ▼
┌──────────────┐┌──────────────┐┌──────────────────────┐
│logger handler││logger handler││LoggerBackends.Handler│
└──────────────┘└──────────────┘└──────────┬───────────┘
                                           │
                                     ┌─────┴─────┐
                                     │           │
                                     ▼           ▼
                                ┌──────────┐┌──────────┐
                                │  logger  ││  logger  │
                                │ backend  ││ backend  │
                                └──────────┘└──────────┘

List of Questionable Advice

1. If you have a pre-1.15 Elixir app on your hands, migrate it to logger handlers.
2. Forget logger backends exist to avoid confusion.
3. Whenever you need to customize your logging stack, try doing it with filters and formatters.
4. If it doesn't work, consider remembering logger backends exist and implement your own to piggyback on built-in overload protection.
5. If you still need to implement a custom logger handler, godspeed 🫡

The conventional wisdom stemming from the famous DRY (Don't Repeat Yourself) principle is that if you notice yourself copying code around, it's a smell. And I agree. Mostly. More often than not. Listen, it's complicated. I think this is a good principle, but even a perfectly functioning analog clock can be wrong for half a year, okay?

One of the few strong opinions I have is that it's not only acceptable but highly desirable to copy code around when you're writing tests. And this has very little to do with the code itself; it's mostly a people thing.

Sorry, tests

Let me tell you a little about myself. Before I discovered Elixir and decided that this language was actually nice enough to make it my full-time job, I was a test engineer. I was testing things for a living. I still wrote code, but that was all test code. And I was writing a lot of it. Not only I, but other test engineers were writing it too. And even regular developers. Sometimes.

When all you write is tests, all your engineering ambitions are funneled into making them good, in a conventional sense. Clean code, SOLID code, and of course, DRY code. But if you work long enough, you at some point face an inevitable question: if our tests are so good, why don't developers write it more often? They certainly have the skills; they deal with much more complex code every day. And this is when I had a simple revelation:

1. People see tests as fundamentally secondary part of the codebase.
2. Dealing with test code is, therefore, a chore that people will actively try to avoid.

This is it. That's the harsh truth. The content of the lib/ folder is just not the same as what's inside the test/ folder.

"But TDD," I hear you saying. "It's very popular; people love tests and see value in them." Of course! A lot of people love writing tests. Writing. Write-time is the honeymoon phase for a test; it's when it's being cared for the most. Unfortunately, it's all downhill from there. They will never see the same treatment again, not even from their own creator.

        ╭──────────────────────────╮
        │ i ain't reading all that │
        ╰──────────────────────────╯
        ╭───────────────────────╮
        │ i'm happy for you tho │
        ╰───────────────────────╯
        ╭────────────────────────╮
        │ or sorry that happened │
        ╰────────────────────────╯
╭────────╮
│ (⌐■_■) │
╰────────╯

Copy copy copy

But what does it have to do with DRY? The thing is, we know that tests are immensely useful, so we need to reduce our own aversion to them as much as possible. And one of the ways to make it easier is to ditch DRY and indulge in self-repetition. Here's an incomplete list of well-intended things people often overuse:

1. Extracting common parts into helper functions
2. Sharing sophisticated setup between tests
3. Using third-party testing libraries with a steep learning curve

Next time you are tempted to create a test helper, think twice. Maybe it makes total sense now. Sure, it's just one click away... No! Stop! People don't open test files because they just feel like doing it. They do it because a test fails, and by this time they're already annoyed. Not only do they not want to read more than one test at a time, they will actively avoid doing so! Every function that's defined somewhere will make their eyes roll. Every setup they need to scroll to will make them angry. And if their rushed fix to a shared setup breaks another test? They may just delete it, because code reviewers have even less incentive to read it.

When writing a test, you have to remember that this is the most effort anybody will ever put into it. Everybody else, including future you, will see this test as a burden. A time capsule with annoying ancient puzzles. So try to fill it with love instead, make a test that's easy to follow, that doesn't require any previous reading or knowledge of that fancy assert_nested_in_any_order custom function used in exactly two places throughout the codebase.

HOWEVER

Of course, you have to exercise your best judgment. I would be crazy to advocate for not having any helpers at all. Take my factories from me, and I'm helpless. Make me create a conn struct without a Phoenix helper, and I don't know how!

In fact, I have a confession to make. Whenever I want to test a large number of cases that can be represented as a test table, I love employing what is usually called parameterized tests:

# Let's say we want to test a function that checks user permissions
# depending on user type and resource privacy

defmodule PermissionsTest do
  use ExUnit.Case

  test_table = [
    {:guest, :public, true},
    {:guest, :private, false},
    {:guest, :system, false},
    {:user, :public, true},
    {:user, :private, true},
    {:user, :system, false},
    {:admin, :public, true},
    {:admin, :private, true},
    {:admin, :system, true},
  ]

  for {user, resource, exp_result} <- test_table do
    test "#{user} access #{resource} resource" do
      user = Users.create_user(%{type: unquote(user)})
      resource = Resources.create_resource(%{privacy: unquote(resource)})

      assert Permissions.has_access?(
        user,
        resource
      ) == unquote(exp_result)
    end
  end
end

As you see, I commit a serious crime of meta-programming in a test, but test tables are just too good. In my defense here, this still reads like a single independent test; it just has a neat little truth table that is right over here in its fullness.

Conclusion

For better or worse, people tend to see test code as less important. And while it may be tempting to fight for higher standards and better tests, I'd argue it's wiser to accept the reality and give both tests and people some slack. It doesn't matter if your test file is 2,000 lines long if you don't have to read it all at once.