.NET ecosystem in Gentoo in year 2023

The Gentoo Dotnet project introduced better support for building .NET-based software using the nuget, dotnet-pkg-base and dotnet-pkg eclasses. This opened new opportunities of bringing new packages depending on .NET ecosystem to the official Gentoo ebuild repository and helping developers that use dotnet-sdk on Gentoo.

New software requiring .NET is constantly being added to the main Gentoo tree, among others that is:

• PowerShell for Linux,
• Denaro — finance application,
• Ryujinx — NS emulator,
• OpenRA — RTS engine for Command & Conquer, Red Alert and Dune2k,
• Pinta — graphics program,
• Pablodraw — Ansi, Ascii and RIPscrip art editor,
• Dafny — verification-aware programming language
• many packages aimed straight at developing .NET projects.

Dotnet project is also looking for new maintainers and users who are willing to help out here and there. Current state of .NET in Gentoo is very good but we can still do a lot better.

Special thanks to people who helped out

• Anna Figueiredo Gomes
• Esteve Varela Colominas
• Andrew Udvare

Links to bugs and announcements

• Bugs
  • gentoo-dotnet–2023/1
  • 900597
• Github PRs
  • 21112
  • 21451
  • 29309
  • 32109
  • 33940
• Active Gentoo .NET projects
  • gentoo-dotnet-guide
  • gentoo-dotnet-maintainer-tools
  • csharp-gentoodotnetinfo
  • eselect-dotnet
  • eselect-pwsh

Building PowerShell

As a part of my work of modernizing the way .NET SDK packages are distributed in Gentoo I delved into packaging a from-source build of PowerShell for Gentoo using the dotnet-pkg eclass.

Packaging pwsh was a little tricky but I got a lot of help from reading the Alpine Linux’s APKBUILD. I had to generate special C# code bindings with ResGen and repackage the PowerShell tarball. Other than this trick, restoring and building PowerShell was pretty straight forward with the NuGet package management support from the dotnet-pkg.eclass.

Alternatively if you do not want to build PowerShell you can install the binary package, I have in plans to keep that package around even after we get the non-binary app-shells/pwsh into the official Gentoo ebuild repository.

Why install modules via Portage?

But why stop on PowerShell when we can also package multiple PS modules?

Installing modules via Portage has many benefits:

• better version control,
• more control over global install,
• no need to enable PS Gallery,
• sandboxed builds,
• using system .NET runtime.

Merging the modules

PowerShell’s method of finding modules is at follows: check paths from the PSModulePath environment variable for directories containing valid .psd1 files which define the PS modules.

By default pwsh tries to find modules in paths:

• user’s modules directory — ~/.local/share/powershell/Modules
• system modules directory in /usr/local — /usr/local/share/powershell/Modules
• Modules directory inside the pwsh home — for example /usr/share/pwsh-7.3/Modules

Because we do not want to touch either /usr/local nor pwsh home, we embed a special environment variable inside the pwsh launcher script to extend the path where pwsh looks for PS modules. The new module directory is located at /usr/share/GentooPowerShell/Modules.

1
2

dotnet-pkg-utils_append_launchervar \
    'PSModulePath="${PSModulePath}:/usr/share/GentooPowerShell/Modules:"'

So every PowerShell module will install it’s files inside /usr/share/GentooPowerShell/Modules.

To follow PS module location convention we add to that path a segment for the real module name and a segment for module version. This also enables us to have proper multi-slotting because most of the time the modules will not block installing other versions.

Take a look at this example from the app-pwsh/posh-dotnet–1.2.3 ebuild:

1
2
3
4
5
6

src_install() {
    insinto /usr/share/GentooPowerShell/Modules/${PN}/${PV}
    doins ${PN}.psd1 ${PN}.psm1

    einstalldocs
}

And that is it. Some packages do not even need to be compiled, they just need files placed into specific location. But when compilation of C# code is needed we have dotnet-pkg to help.

Binpkgs generated by user

The binary packages generated by user can have architecture-specific optimizations because they are generated after they were compiled by the host Portage installation.

In addition binpkgs are generated from ebuilds so if there is a USE flag incompatibility on the consumer system then the binpkg will not be installed on the host and Portage will fall back to from-source compilation.

Those binary packages can use two formats: XPAK and GPKG.

XPAK had many issues and is getting superseded by the GPKG format. Beware of upcoming GPKG transition and if you must use XPAKs then you should explicitly enable it in your system’s Portage configuration.

To host a binary package distribution server see the Binary package guide on the Gentoo wiki.

Bin packages in a repository

Binary packages in ::gentoo (the official Gentoo repository) have the -bin suffix.

Those packages might have USE flags but generally they are very limited in case of customizations or code optimizations because they were compiled either by a Gentoo developer or by a given package upstream maintainer (or their CI/CD system).

Those packages land in ::gentoo mostly because it is too hard (or even impossible) to compile them natively by Portage. Most of the time those packages use very complicated build systems or do not play nice with network sandbox like (e.g. Scala-based projects) or use very large frameworks/libraries like (e.g. Electron).

They can also be added to the repository because they are very desirable either by normal users (e.g. www-client/firefox-bin) or for (from-source) package bootstrapping purposes (e.g. dev-java/openjdk-bin). Such packages are sometimes generated from the regular source packages inside ::gentoo and later repackaged.

Portage

Configure the following for Portage.

1

dev-util/pkgcheck emacs

Emerge

Emerge the following packages:

• app-emacs/company-ebuild
• dev-util/pkgcheck

Company-Ebuild should pull in app-emacs/ebuild-mode, if that does not happen, then report a bug ;-D

Standard

Add the following to your user's Emacs initialization file. The initialization file is either ~/.emacs.d/init.el or ~/.config/emacs/init.el for newer versions of GNU Emacs.

1
2
3
4
5
6
7
8

(require 'ebuild-mode)
(require 'company-ebuild)
(require 'flycheck)
(require 'flycheck-pkgcheck)

(add-hook 'ebuild-mode-hook 'company-ebuild-setup)
(add-hook 'ebuild-mode-hook 'flycheck-mode)
(add-hook 'ebuild-mode-hook 'flycheck-pkgcheck-setup)

Use-Package

We can also configure our environment using a use-package macro that simplifies the setup a little bit.

To use the below configuration the app-emacs/use-package package will have to be installed.

1
2
3
4
5
6
7
8
9

(require 'use-package)

(use-package ebuild-mode
  :defer t
  :mode "\\.\\(ebuild\\|eclass\\)\\'"
  :hook
  ((ebuild-mode . company-ebuild-setup)
   (ebuild-mode . flycheck-mode)
   (ebuild-mode . flycheck-pkgcheck-setup)))

The :defer t and :mode "..." enable deferred loading which theoretically speeds up GNU Emacs initialization time at the cost of running the whole use-package block of ebuild-mode configuration when the :mode condition is met.

Prototype

Recently while browsing the Alpine git repo I noticed they have a function called snapshot, see: https://git.alpinelinux.org/aports/tree/testing/dart/APKBUILD#n45 I am not 100% sure about how that works but a wild guess is that the developers can run that function to fetch the sources and maybe later upload them to the Alpine repo or some sort of (cloud?) storage.

In Portage there exists a pkg_config function used to run miscellaneous configuration for packages. The only major difference between src_snapshot and that would of course be that users would never run snapshot.

Sandbox

Probably only the network sandbox would have to be lifted out… to fetch the sources of course.

But also a few (at least one?) special directories and variables would be useful.

Potential benefits

Running tests

• test BEFORE (src_test) and AFTER (pkg_postinst) installation
• test if and how services break if they are not reloaded
• test buildsystem configuration
• sandbox enforces strict and consistent build rules
• benchmarking with different compilation flags and libraries versions/releases

Configuration matrix

We can test across Cartesian product of different configuration settings, like:

• USE flags
• MAKEOPTS
• CFLAGS, CXXFLAGS, CPPFLAGS, LDFAGS, RUSTFLAGS, etc.
• arches (cross-compilation or run in qemu)
• static linking
• supported releases & versions of libraries (eg. glibc & musl)

Also, we could create diffs of installed files across different merges.

Reproducibility

• mini overlay with ::gentoo or any other (eg. company's own) as master
• record VCS (eg. git) hash of the dependent overlays

Binaries

• grab dependencies from binhosts
• distribute built binaries (maybe upload to a company's own artifacts server)
• make AppImages

Getting there

How do we run this?

Do we want to write a proper tool, which we probably do or do we just run Portage + shells scripts?

Do we want to run under root, user, in eprefix, maybe all in docker?

Configuration files

The .portci directory contains the configuration.

Bug 799626

Link: bugs.gentoo.org/799626

Instead of using Ansible, Python, Yaml or Scheme we might use something similar to this for simple configuration, or if gets merged to upstream Portage the better.

Worth mentioning is the idea from Michał Górny who proposes to configure portage with toml files, like the example given in the bug report.

1
2
3
4
5
6
7
8

[package.unmask]
~virtual/libcrypt-2

[package.use.mask]
sys-libs/libxcrypt -system -split-usr

[package.use.force]
sys-libs/glibc -crypt

Also, package.x + Toml == a match made in heaven, it looks very nice!

Switching from a git repository hosting

For nearly 2 years I have been synchronizing settings between my Gentoo machines via a git repository that is installed system-wide via portage itself.

This actually gets a little tiresome and slow as the package that installs the configuration has to be re-installed each time we want to update the settings.

So I thought it could be cool if I can just push files around between my machines with one command. Now: we can use rsync or scp but /etc/portage is owned by root so we either have to change the permissions to a user we use to ssh or a group that user is in.

Portage users

First change portage permissions of /etc/portage, now all users in the portage group will be able to modify the configuration

1
2

chown -R root:portage /etc/portage
chmod -R g+w /etc/portage

Synchronizing

We can do a rsync from different machine with:

1

rsync -r /etc/portage REMOTEHOST:/etc --exclude=".git" --exclude="make.profile" --copy-unsafe-links

Reminder that this can also be done as a cron job!

Portage snapshots with git

Probably the best thing I got out of genlica is that I engineered a script to automatically do a git commit of the changes to the portage configuration.

Emerge git

1

emerge --noreplace --verbose dev-vcs/git

Copy the file that does the snapshots

1
2

mkdir -p /etc/portage/postsync.d
curl -o /etc/portage/postsync.d/99-degitmerge.sh "https://gitlab.com/xgqt/genlica/-/raw/master/portage/postsync.d/99-degitmerge.sh"