François Scala

François Scala arcenik

Switzerland
https://francois.scala.name/

Organizations

Perfomance on ollama 0.24.0

Using gemma4.31b, it runs 100% on the GPU

NAME          ID              SIZE     PROCESSOR    CONTEXT    UNTIL
gemma4:31b    6316f0629137    47 GB    100% GPU     262144     4 minutes from now

And using a simple question `what is json ?’ I obtained an answer at ~10 token per second.

total duration:       1m8.796615034s
load duration:        164.085088ms
prompt eval count:    17 token(s)
prompt eval duration: 205.756478ms
prompt eval rate:     82.62 tokens/s
eval count:           702 token(s)
eval duration:        1m8.1488854s
eval rate:            10.30 tokens/s

Performance on ollama v0.30.5

After upgrading to v0.30.5, ollama-rocm now detect both the CPU and the GPU

jui 15 19:18:57 phi25 ollama[1548]: device_info:
jui 15 19:18:57 phi25 ollama[1548]:   - CPU     : AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (31727 MiB, 31727 MiB free)
jui 15 19:18:57 phi25 ollama[1548]:   - ROCm0   : AMD Radeon 8060S Graphics (98304 MiB, 29071 MiB free)

And now, the modele is spread across the two devices

NAME          ID              SIZE     PROCESSOR          CONTEXT    UNTIL
gemma4:31b    6316f0629137    34 GB    57%/43% CPU/GPU    262144     4 minutes from now

And this has a major impact on perfomance, as the token per second is almsot devided by two.

total duration:       1m44.78409003s
load duration:        309.925333ms
prompt eval count:    17 token(s)
prompt eval duration: 770.637ms
prompt eval rate:     22.06 tokens/s
eval count:           689 token(s)
eval duration:        1m43.701536s
eval rate:            6.64 tokens/s

An issue about this is opened on github: #16462

Running Ollama on Nixos on Strix Halo
How I run Ollama on the last addition to my homelab, the MS S1 Max from Minisforum featuring the Strix Halo processor from AMD, 128Gb unified memory and 2 SFP+ 10Gbe network interfaces.
NixOS
Being a long time Debian user, I want to try out this declarative and reproductible system NixOS.
Install base system
After installing NixOS with just console, I added a few packages, starting by ssh.
Extract from /etc/nixos/configuration.nix:
environment.systemPackages = with pkgs; [ amdgpu_top bmon htop nvd tmux vim ];
Configure Ollama
Add a dedicated confuration file in /etc/nixos, /etc/nixos/ollama.nix:

Extracting MongoDB from the Unifi container

For many years, I’ve ran my Unifi network controller with Docker Compose using included mongodb server.

But now it is time to change this. To externalize and upgrade it.

Previous situation

Until now, Unifi was deployed using the included MongoDB server.

  unifi:
    image: goofball222/unifi:10.0.160-ubuntu
    hostname: unifi
    user: unifi
    restart: always
    ports:
      - 3478:3478/udp # STUN connection
      - 6789:6789 # throughput measurement from Android/iOS app
      - 8080:8080 # UAP/USW/USG to inform controller
      - 8443:8443 # controller GUI / API
      - 8880:8880 # HTTP portal redirect
      - 8843:8843 # HTTPS portal redirect
      - 10001:10001/udp # UBNT discovery broadcasts
    environment:
      - DB_MONGO_LOCAL=true
    volumes:
      - /srv/unifi/data:/usr/lib/unifi/data
      - /srv/unifi/log:/usr/lib/unifi/log
      - /srv/unifi/cert:/usr/lib/unifi/cert

Migration plan

Check Unifi backup
Stop the Unifi container
Copy/move the MongoDB folder to the new location
Start the MongoDB container, using the same version
Update Unifi Docker Compose to use the external MongoDB
Start the Unifi container

Check Unifi backup

# cd /srv
# ls -lh unifi/data/backup/

Stop the Unifi container

$ docker compose down unifi

Copy the data

# cd /srv
# cp -rp unifi/data/db mongounifi

New mongounifi service

Using the same version as in the Unifi container, to be upgraded later.

Write a Prometheus exporter in Go

How to expose Prometheus metrics from a json blob returned by a server.

Here is a walk through, based on json retreived from an EthSwarm node.

Retreive the json from EthSwarm status API

Here is a sample for the status json, from the API documentation,

{
  "overlay": "36b7efd913ca4cf880b8eeac5093fa27b0825906c600685b6abdd6566e6cfe8f",
  "proximity": 0,
  "beeMode": "light",
  "reserveSize": 0,
  "reserveSizeWithinRadius": 0,
  "pullsyncRate": 0,
  "storageRadius": 0,
  "connectedPeers": 0,
  "neighborhoodSize": 0,
  "requestFailed": true,
  "batchCommitment": 0,
  "isReachable": true,
  "lastSyncedBlock": 0,
  "committedDepth": 0
}

Prometheus exporter package

ethswarm/status.go start with the package declaration and imports

package ethswarm

import (
	"encoding/json"
	"log"
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
)

Observations and analysis about systemd-networkd-wait-online.service

Since a few time, I have to wait two minutes on each reboot, waiting for a faulty systemd service: systemd-networkd-wait-online.service

server# journalctl --boot -u systemd-networkd-wait-online.service
Jun 22 13:38:00 server systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Jun 22 13:40:00 server systemd-networkd-wait-online[472]: Timeout occurred while waiting for network connectivity.
Jun 22 13:40:00 server systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jun 22 13:40:00 server systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Jun 22 13:40:00 server systemd[1]: Failed to start systemd-networkd-wait-online.service - Wait for Network to be Configured.

The service is linked to the network-online systemd target.

server# ls -l /etc/systemd/system/network-online.target.wants/
total 4
lrwxrwxrwx 1 root root 42 Mar 23 12:02 networking.service -> /usr/lib/systemd/system/networking.service
lrwxrwxrwx 1 root root 60 Apr 26 18:26 systemd-networkd-wait-online.service -> /usr/lib/systemd/system/systemd-networkd-wait-online.service

Identify the problem

Reproduce the problem, running the command without any additional argument. Real time is 2 minutes and return 1.

server# time /usr/lib/systemd/systemd-networkd-wait-online ; echo $?
Timeout occurred while waiting for network connectivity.

real    2m0.236s
user    0m0.004s
sys     0m0.011s
1

Attempt to fix

Testing the commmand with extra argument, the command exit immediately and returns 0.

server# time /usr/lib/systemd/systemd-networkd-wait-online --timeout=2 -i enp2s0f0 ; echo $?

real    0m0.017s
user    0m0.013s
sys     0m0.004s
0

Docker as Build Environment
You have a small/mid sized server and you want to install new software or custom kernel on it. But it take forever to build anything on it compared to your brand new 8 or 12 core modern laptop with fast NVMe SSD.
In addition, you don´t want to pollute your server with all the build dependencies.
Docker is here to save the day, allowing you to create fast and disposable build environments.
-rw-r--r-- 1 fs fs 8.6M 2024-12-22 17:33 linux-headers-6.12.6-test_6.12.6-1_amd64.deb -rw-r--r-- 1 fs fs 19M 2024-12-22 17:33 linux-image-6.12.6-test_6.12.6-1_amd64.deb -rw-r--r-- 1 fs fs 285M 2024-12-22 17:33 linux-image-6.12.6-test-dbg_6.12.6-1_amd64.deb -rw-r--r-- 1 fs fs 1.4M 2024-12-22 17:33 linux-libc-dev_6.12.6-1_amd64.deb
Here are two examples with linux kernel image and zfs-linux backport.