Common Project Layout, version 0

:: programming, software engineering

By: Maciej Barć

This is a tongue-in-cheek “draft” for Common Project Layout, version 0. It will probably never become any sort of adopted standard but I think it is good to share some ideas that I had while working on this.

Definition

Common Project Layout (CPL) is a set of good practices for structuring medium-to-large monorepo-like software repositories.

Benefits

CPL helps with code organization. It can be a good “framework” (in a very loose meaning of this word) to modularize product components.

It can make large repositories easier to work with and understand.

Upfront limitations

CPL is strictly designed for “hosting” software and all the non-code assets are left up to the engineers to decide their location.

For example branding assets could be put into the Branding top-level directory, but on the other hand are we sure they will stay the same with major version?

Since we can agree that we consider documentation “producers” (not the produced artifacts) to be code we could also acknowledge that some assets could have their own versioned subproject.

Requirements

Versioning

CPL requires that the software is versioned inside directories whose names include the version. Recommended pattern is to name directories vMAJOR where MAJOR is either the current tagged major version or one that will be if no tags exist. It is also recommended to group the vMAJOR directories under one common directory, for example Source.

Subprojects

The vMAJOR could theoretically contain all the source code mixed together but it should be grouped and organized by their purpose.

Subproject is defined as a directory inside a versioned (vMAJOR) directory. “Versioned subproject” and “subproject” are synonymous to CPL.

To mark the purpose of a subproject, whether it is to be used as a helper or as a “container” for source that is actually exposed (or binaries created from it), it should be adequately named.

For helpers name does not matter but for source subproject it should be prefixed by project name.

For example we could have this layout:

1
2
3
4
5
6
7
8
 
Source/
└── v1/
    ├── Makefile
    ├── VERSION
    ├── admin/
    ├── make/
    ├── my-project-app/
    └── my-project-util/

In the above example my-project-app and my-project-lib are the source subproject and admin and make are subproject that are there only to help in building, managing and deploying the actual source subprojects.

At the and it is up to the engineer to choose if something is considered a source subproject. For example: If we have a helper subproject that all it does is hold Docker / Podman files for creating a development container what should we name it? As of now I had named them PROJECT-dev-container.

Recommendations

Make and admin

I think it is a good practice for each vMAJOR to have a Makefile, or equivalent in other build system, that will call scripts inside vMAJOR/admin directory that each take care of some small / specific task.

For example the vMAJOR/Makefile recipe for build can call admin/build_my_project_app.py and admin/build_my_project_lib.py. Each those scripts would call the “real” build system specific to the subproject they act upon.

VERSION file

It is nice to have a VERSION file in the vMAJOR directory. It can be reused by build tools and also to show what was the last version worked upon inside vMAJOR, the latest git tag can either be put on different major version or simply not be there yet.

References

See those repositories for referencing the CPL layout:

Using gzexe for shipping Racket executables

:: lisp, racket, scheme, tutorial

By: Maciej Barć

Racket executables made by raco exe are known to be quite large. One of tools that can be used to help reduce the size of produced binaries is the gzexe program.

gzeze is a tool that can compress a executable binary. It can be acquired by installing gzip on most Linux distributions (included in the app-arch/gzip package on Gentoo).

Creating a hello-world executable with Racket

Write following contents to hello-world.rkt file:

1
2
3
4
5
6
7
 
#lang racket/base

(define (main)
  (displayln "It REPLs. Ship it!"))

(module+ main
  (main))

To make a binary run:

1
 
raco exe --orig-exe -v -o hello-world hello-world.rkt

The file hello-world will be produced.

This is what file hello-world says about it:

1
2
3
 
hello-world: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
for GNU/Linux 3.2.0, stripped

This “small” executable weights 46 MB!

In comparison busybox weights around 2 MB.

Compressing with gzexe

Keep in mind that gzeze will overwrite the compressed file and create a backup with appended "~".

1
 
gzexe hello-world

And this gives us only 8,5 MB. Nice!

In comparison bazel, which is a single-binary build system written in JAVA, executable takes 33 MB on my Gentoo machine. I tried compressing it with gzexe and it reduces it only by 10%, to around 29 MB.

gzexeis not a silver bullet but with Racket exes it works very nicely.

Genkernel in 2023

:: gentoo, kernel, linux, sysadmin, system, tutorial

By: Maciej Barć

I really wanted to look into the new kernel building solutions for Gentoo and maybe migrate to dracut, but last time I tried, ~1.5 years ago, the initreamfs was now working for me.

And now in 2023 I’m still running genkernel for my personal boxes as well as other servers running Gentoo.

I guess some short term solutions really become defined tools :P

So this is how I rebuild my kernel nowadays:

  1. Copy old config

    1
2
    		 
    cd /usr/src
cp linux-6.1.38-gentoo/.config linux-6.1.41-gentoo/

  2. Remove old kernel build directories

    1
    		 
    rm -r linux-6.1.31-gentoo

  3. Run initial preparation

    1
    		 
    ( eselect kernel set 1 && cd /usr/src/linux && make olddefconfig )

  4. Call genkernel

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    		 
    genkernel                                                       \
    --no-menuconfig                                             \
    --no-clean                                                  \
    --no-clear-cachedir                                         \
    --no-cleanup                                                \
    --no-mrproper                                               \
    --lvm                                                       \
    --luks                                                      \
    --mdadm                                                     \
    --nfs                                                       \
    --kernel-localversion="-$(hostname)-$(date '+%Y.%m.%d')"    \
    all

  5. Rebuild the modules

    If in your /etc/genkernel.conf you have MODULEREBUILD turned off, then also call emerge:

    1
    		 
    emerge -1 @module-rebuild