Posts tagged gentoo

Ebuild lit tests

:: gentoo, ebuild, tutorial, python

By: Maciej Barć

Patching

The file lit.site.cfg has to be inspected for any incorrect calls to executables. For example see src_prepare function form dev-lang/boogie.

Eclasses

Because we will need to specify how many threads should lit run we need to inherit multiprocessing to detect how many parallel jobs the portage config sets.

1
 
inherit multiprocessing

Dependencies

Ensure that dev-python/lit is in BDEPEND, but also additional packages may be needed, for example dev-python/OutputCheck.

1
2
3
4
5
6
7
 
BDEPEND="
    ${RDEPEND}
    test? (
        dev-python/lit
        dev-python/OutputCheck
    )
"

Bad tests

To deal with bad test you can simply remove the files causing the failures.

1
2
3
4
5
6
7
8
9
 
local -a bad_tests=(
    civl/inductive-sequentialization/BroadcastConsensus.bpl
    civl/inductive-sequentialization/PingPong.bpl
    livevars/bla1.bpl
)
local bad_test
for bad_test in ${bad_tests[@]} ; do
    rm "${S}"/Test/${bad_test} || die
done

Test phase

--threads $(makeopts_jobs) specifies how many parallel tests to run.

--verbose option will show output of failed tests.

Last lit argument specifies where lit should look for lit.site.cfg and tests.

1
2
3
 
src_test() {
    lit --threads $(makeopts_jobs) --verbose "${S}"/Test || die
}

Ebuild-mode

:: gentoo, portage, ebuild, emacs, tutorial, pkgcheck

By: Maciej Barć

Portage

Configure the following for Portage.

1
 
dev-util/pkgcheck emacs

Emerge

Emerge the following packages:

  • app-emacs/company-ebuild
  • dev-util/pkgcheck

Company-Ebuild should pull in app-emacs/ebuild-mode, if that does not happen, then report a bug ;-D

Standard

Add the following to your user's Emacs initialization file. The initialization file is either ~/.emacs.d/init.el or ~/.config/emacs/init.el for newer versions of GNU Emacs.

1
2
3
4
5
6
7
8
 
(require 'ebuild-mode)
(require 'company-ebuild)
(require 'flycheck)
(require 'flycheck-pkgcheck)

(add-hook 'ebuild-mode-hook 'company-ebuild-setup)
(add-hook 'ebuild-mode-hook 'flycheck-mode)
(add-hook 'ebuild-mode-hook 'flycheck-pkgcheck-setup)

Use-Package

We can also configure our environment using a use-package macro that simplifies the setup a little bit.

To use the below configuration the app-emacs/use-package package will have to be installed.

1
2
3
4
5
6
7
8
9
 
(require 'use-package)

(use-package ebuild-mode
  :defer t
  :mode "\\.\\(ebuild\\|eclass\\)\\'"
  :hook
  ((ebuild-mode . company-ebuild-setup)
   (ebuild-mode . flycheck-mode)
   (ebuild-mode . flycheck-pkgcheck-setup)))

The :defer t and :mode "..." enable deferred loading which theoretically speeds up GNU Emacs initialization time at the cost of running the whole use-package block of ebuild-mode configuration when the :mode condition is met.

src_snapshot

:: ebuild, gentoo, portage, prototype

By: Maciej Barć

Prototype

Recently while browsing the Alpine git repo I noticed they have a function called snapshot, see: https://git.alpinelinux.org/aports/tree/testing/dart/APKBUILD#n45 I am not 100% sure about how that works but a wild guess is that the developers can run that function to fetch the sources and maybe later upload them to the Alpine repo or some sort of (cloud?) storage.

In Portage there exists a pkg_config function used to run miscellaneous configuration for packages. The only major difference between src_snapshot and that would of course be that users would never run snapshot.

Sandbox

Probably only the network sandbox would have to be lifted out… to fetch the sources of course.

But also a few (at least one?) special directories and variables would be useful.

Pkgcheck-Flycheck

:: emacs, pkgcheck, gentoo, ebuild, tutorial

By: Maciej Barć

News

Repository

With this commit first GNU Emacs integration was merged into the pkgcheck repository.

History

Thanks

Huge thanks to Sam James and Arthur Zamarin for support and interest in getting this feature done.

Installation

Unmasking

The Flycheck integration is unreleased as of now, this will (hopefully) change in the future, but for now You need live versions of snakeoil, pkgcore and pkgcheck.

File: /etc/portage/package.accept_keywords/pkgcore.conf

1
2
3
 
dev-python/snakeoil  **
sys-apps/pkgcore     **
dev-util/pkgcheck    **

Also You will need to unmask app-emacs/flycheck and its dependencies.

File: /etc/portage/package.accept_keywords/emacs.conf

1
2
3
 
app-emacs/epl
app-emacs/pkg-info
app-emacs/flycheck

Emerging

Install pkgcheck with the emacs USE flag enabled.

File: /etc/portage/package.use/pkgcore.conf

1
 
dev-util/pkgcheck    emacs

Afterwards run:

1
2
 
emerge -1av dev-python/snakeoil sys-apps/pkgcore dev-util/pkgcheck
emerge -av --noreplace dev-util/pkgcheck

Configuration

Following is what I would suggest to put into your Emacs config file:

1
2
3
4
5
6
7
8
 
(require 'ebuild-mode)
(require 'flycheck)
(require 'flycheck-pkgcheck)

(setq flycheck-pkgcheck-enable t)

(add-hook 'ebuild-mode-hook 'flycheck-mode)
(add-hook 'ebuild-mode-hook 'flycheck-pkgcheck-setup)

If You are using use-package:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
 
(use-package flycheck
  :ensure nil)

(use-package ebuild-mode
  :ensure nil
  :hook ((ebuild-mode . flycheck-mode)))

(use-package flycheck-pkgcheck
  :ensure nil
  :custom ((flycheck-pkgcheck-enable t))
  :hook ((ebuild-mode . flycheck-pkgcheck-setup)))

The lines with :ensure nil are there to prevent use-package from trying to download the particular package from Elpa (because we use system packages for this configuration).

Portage CI

:: continuous integration, gentoo, portage, quality assurance, testing

By: Maciej Barć

Potential benefits

Running tests

  • test BEFORE (src_test) and AFTER (pkg_postinst) installation
  • test if and how services break if they are not reloaded
  • test buildsystem configuration
  • sandbox enforces strict and consistent build rules
  • benchmarking with different compilation flags and libraries versions/releases

Configuration matrix

We can test across Cartesian product of different configuration settings, like:

  • USE flags
  • MAKEOPTS
  • CFLAGS, CXXFLAGS, CPPFLAGS, LDFAGS, RUSTFLAGS, etc.
  • arches (cross-compilation or run in qemu)
  • static linking
  • supported releases & versions of libraries (eg. glibc & musl)

Also, we could create diffs of installed files across different merges.

Reproducibility

  • mini overlay with ::gentoo or any other (eg. company's own) as master
  • record VCS (eg. git) hash of the dependent overlays

Binaries

  • grab dependencies from binhosts
  • distribute built binaries (maybe upload to a company's own artifacts server)
  • make AppImages

Getting there

How do we run this?

Do we want to write a proper tool, which we probably do or do we just run Portage + shells scripts?

Do we want to run under root, user, in eprefix, maybe all in docker?

Configuration files

The .portci directory contains the configuration.

Bug 799626

Link: bugs.gentoo.org/799626

Instead of using Ansible, Python, Yaml or Scheme we might use something similar to this for simple configuration, or if gets merged to upstream Portage the better.

Worth mentioning is the idea from Michał Górny who proposes to configure portage with toml files, like the example given in the bug report.

1
2
3
4
5
6
7
8
 
[package.unmask]
~virtual/libcrypt-2

[package.use.mask]
sys-libs/libxcrypt -system -split-usr

[package.use.force]
sys-libs/glibc -crypt

Also, package.x + Toml == a match made in heaven, it looks very nice!

Portage

:: gentoo, portage

By: Maciej Barć

Switching from a git repository hosting

For nearly 2 years I have been synchronizing settings between my Gentoo machines via a git repository that is installed system-wide via portage itself.

This actually gets a little tiresome and slow as the package that installs the configuration has to be re-installed each time we want to update the settings.

So I thought it could be cool if I can just push files around between my machines with one command. Now: we can use rsync or scp but /etc/portage is owned by root so we either have to change the permissions to a user we use to ssh or a group that user is in.

Portage users

First change portage permissions of /etc/portage, now all users in the portage group will be able to modify the configuration

1
2
 
chown -R root:portage /etc/portage
chmod -R g+w /etc/portage

Synchronizing

We can do a rsync from different machine with:

1
 
rsync -r /etc/portage REMOTEHOST:/etc --exclude=".git" --exclude="make.profile" --copy-unsafe-links

Reminder that this can also be done as a cron job!

Portage snapshots with git

Probably the best thing I got out of genlica is that I engineered a script to automatically do a git commit of the changes to the portage configuration.

Emerge git

1
 
emerge --noreplace --verbose dev-vcs/git

Copy the file that does the snapshots

1
2
 
mkdir -p /etc/portage/postsync.d
curl -o /etc/portage/postsync.d/99-degitmerge.sh "https://gitlab.com/xgqt/genlica/-/raw/master/portage/postsync.d/99-degitmerge.sh"

Sage Math on Gentoo

:: gentoo, tutorial, mathematics

By: Maciej Barć

Intro

Sage may be available on your distro but on Gentoo such frivolities for students are not there yet, so I had to install it the manual way.

User

I went to the Sage website, to the "download-source" link. The source mirror I picked was France.

Ok, so let's follow Sage Math build instructions and get it going

1
2
3
4
5
 
wget www-ftp.lip6.fr/pub/math/sagemath/src/sage-9.2.tar.gz
tar xvf sage-9.2.tar.gz
cd sage-9.2
./configure
make

And now, let's wait…

Portage

After a long, long, long time waiting I remembered that there existed a Gentoo overlay for Sage. And in the meantime I thought I'd try that solution instead since some good Gentoo people already did most of the effort.

Just a few files to edit…

File: /etc/portage/repos.conf/sage-on-gentoo.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 
# -*- conf -*-


[sage-on-gentoo]

auto-sync = yes
location = /var/db/repos/sage-on-gentoo
priority = 999
sync-git-clone-extra-opts = --depth=999999999 --no-shallow-submodules --verbose
sync-git-pull-extra-opts = --verbose
sync-type = git
sync-umask = 022
sync-uri = https://github.com/cschwan/sage-on-gentoo.git
sync-user = root:portage

File: /etc/portage/package.accept~keywords~/zz-sage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
 
# -*- conf -*-


sci-mathematics/sage                    **

*/*::sage-on-gentoo

dev-python/cvxopt

media-gfx/tachyon

sci-libs/bliss
sci-libs/dsdp
sci-libs/fflas-ffpack
sci-libs/fplll
sci-libs/libhomfly
sci-libs/linbox
sci-libs/m4rie

sci-mathematics/glpk

File: /etc/portage/package.use/zz-sage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
 
# -*- conf -*-


sci-mathematics/sage                    -doc-html -doc-html-bin
sci-mathematics/sage                    -doc-pdf -doc-pdf-bin
sci-mathematics/sage                    -jmol
sci-mathematics/sage                    X bliss meataxe

dev-python/pplpy                        doc

sci-libs/cddlib                         tools
sci-libs/pynac                          -giac

sci-mathematics/eclib                   flint
sci-mathematics/flint                   ntl
sci-mathematics/glpk                    gmp
sci-mathematics/gmp-ecm                 -openmp
sci-mathematics/lcalc                   pari
sci-mathematics/maxima                  ecls
sci-mathematics/pari                    gmp doc

Now - let's build Sage with Portage!

>>> Emerging (1 of 100) sci-mathematics/cliquer–1.21::gentoo

At least I know more or less how long I'm going to wait and know what exactly fails to build, if anything does.

Final

Build

$ qlop -tv sage

> 2021–03–23T20:49:12 >>> sci-mathematics/sage–9999: 55′04″

Git stats

Overlays

This info we can easily gather with executing:

1
 
emerge --info

Gentoo' HEAD: 33f2d770c28307b1e9a1199c681e1c543602c6d4

Sage-on-Gentoo's HEAD: f7eac5b7e1a844132164b7593dab85cd87918664

Sage

Sage repository's HEAD (because we are using the live (9999) ebuild):

1
 
cat /var/cache/distfiles/git3-src/sagemath_sage.git/refs/heads/develop

Which returns: 5cb72aade9b297c10bb0f1ae8529466e5b5eb41d