mirror/guix
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/guix.git synced 2025-01-18 21:46:35 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
No description
152486 commits 90 branches 36 tags 690 MiB
Scheme 98.1%
C++ 0.9%
Shell 0.5%
Makefile 0.3%
Find a file
Tomas Volf 50927338dd
gexp: Improve support of Unicode characters. 
Support for non-ASCII characters was mixed.  Some gexp forms did support them,
while others did not.  Combined with current value for
%default-port-conversion-strategy, that sometimes led to unpleasant surprises.
For example:

    (scheme-file "utf8" #~(with-output-to-file #$output
                            (λ _ (display "猫"))))

Was written to the store as:

    ((? _ (display "\u732b")))

No, that is not font issue on your part, that is an actual #\? instead of the
lambda character.  Which, surprisingly, does not do what it should when
executed.

The solution is to switch to C.UTF-8 locale where possible, since it is now
always available.  Or to explicitly set the port encoding.

No tests are provided, since majority of tests/gexp.scm use guile in version
2, and it tends to work under it.  The issues occur mostly with guile 3.

I did test it locally using:

      #!/bin/sh
      set -eu
      set -x

      [ -f guix.scm ] || { echo >&2 Run from root of Guix repo.; exit 1; }
      [ -f gnu.scm  ] || { echo >&2 Run from root of Guix repo.; exit 1; }

      cat >猫.scm <<'EOF'
      (define-module (猫)
        #:export (say))

      (define (say)
        "nyaaaa~~~~!")
      EOF

      mkdir -p dir-with-utf8-file
      cp 猫.scm dir-with-utf8-file/

      cat >repro.scm <<'EOF'
      (use-modules (guix build utils)
                   (guix derivations)
                   (guix gexp)
                   (guix store)
                   (ice-9 ftw)
                   (ice-9 textual-ports))

      (define cat "猫")

      (define (drv-content drv)
        (call-with-input-file (derivation->output-path drv)
          get-string-all))

      (define (out-content out)
        (call-with-input-file out
          get-string-all))

      (define (drv-listing drv)
        (scandir (derivation->output-path drv)))

      (define (dir-listing dir)
        (scandir dir))

      (define-macro (test exp lower? report)
        (let ((type (car exp)))
          `(false-if-exception
            (let ((drv (with-store %store
                         (run-with-store %store
                           (,(if lower? lower-object identity) ,exp)))))
              (format #t "~%~a:~%" ',type)
              (when (with-store %store
                      (build-derivations %store (list drv)))
                (format #t "~a~%" (,report drv)))))))

      (test (computed-file "utf8"
                           #~(with-output-to-file #$output
                               (λ _ (display #$cat))))
            #t drv-content)

      (test (program-file "utf8"
                          #~((λ _ (display #$cat))))
            #t drv-content)

      (test (scheme-file "utf8"
                         #~((λ _ (display #$cat))))
            #t drv-content)

      (test (text-file* "utf8" cat cat cat)
            #f drv-content)

      (test (compiled-modules '((猫)))
            #f drv-listing)

      (test (file-union "utf8" `((,cat ,(plain-file "utf8" cat))))
            #t drv-listing)

      ;;; No fix needed:
      (test (imported-modules '((猫)))
            #f dir-listing)

      (test (local-file "dir-with-utf8-file" #:recursive? #t)
            #t dir-listing)

      (test (plain-file "utf8" cat)
            #t out-content)

      (test (mixed-text-file "utf8" cat cat cat)
            #t drv-content)

      (test (directory-union "utf8" (list (local-file "dir-with-utf8-file"
                                                      #:recursive? #t)))
            #t dir-listing)
      EOF

      guix shell -CWN -D guix glibc-locales -- \
           env LANG=C.UTF-8 ./pre-inst-env guix repl -- ./repro.scm

Before this commit, the output is:

      + '[' -f guix.scm ']'
      + '[' -f gnu.scm ']'
      + cat
      + mkdir -p dir-with-utf8-file
      + cp 猫.scm dir-with-utf8-file/
      + cat
      + guix shell -CWN -D guix glibc-locales -- env LANG=C.UTF-8 ./pre-inst-env guix repl -- ./repro.scm

      computed-file:
      ?

      program-file:
      #!/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile --no-auto-compile
      !#
      ((? _ (display "\u732b")))

      scheme-file:
      ((? _ (display "\u732b")))

      text-file*:
      ???

      compiled-modules:
      building path(s) `/gnu/store/ay3jifyvliigfgnz67jf0kgngzpya5a5-module-import-compiled'
      Backtrace:
                 5 (primitive-load "/gnu/store/rn7b0dq6iqfmmqyqzamix2mjmfy?")
      In ice-9/eval.scm:
          619:8  4 (_ #f)
      In srfi/srfi-1.scm:
         460:18  3 (fold #<procedure 7ffff79245e0 at ice-9/eval.scm:336:1?> ?)
      In ice-9/eval.scm:
         245:16  2 (_ #(#(#<directory (guix build utils) 7ffff779f320>) # ?))
      In ice-9/boot-9.scm:
        1982:24  1 (_ _)
      In unknown file:
                 0 (stat "./???.scm" #<undefined>)

      ERROR: In procedure stat:
      In procedure stat: No such file or directory: "./???.scm"
      builder for `/gnu/store/dxg87135zcd6a1c92dlrkyvxlbhfwfld-module-import-compiled.drv' failed with exit code 1

      file-union:
      (. .. ?)

      imported-modules:
      (. .. 猫.scm)

      local-file:
      (. .. 猫.scm)

      plain-file:
      猫

      mixed-text-file:
      猫猫猫

      directory-union:
      (. .. 猫.scm)

Which I think you will agree is far from optimal.  After my fix the output
changes to:

      + '[' -f guix.scm ']'
      + '[' -f gnu.scm ']'
      + cat
      + mkdir -p dir-with-utf8-file
      + cp 猫.scm dir-with-utf8-file/
      + cat
      + guix shell -CWN -D guix glibc-locales -- env LANG=C.UTF-8 ./pre-inst-env guix repl -- ./repro.scm

      computed-file:
      猫

      program-file:
      #!/gnu/store/8kbmn359jqkgsbqgqxnmiryvd9ynz8w7-guile-3.0.9/bin/guile --no-auto-compile
      !#
      ((λ _ (display "猫")))

      scheme-file:
      ((λ _ (display "猫")))

      text-file*:
      猫猫猫

      compiled-modules:
      (. .. 猫.go)

      file-union:
      (. .. 猫)

      imported-modules:
      (. .. 猫.scm)

      local-file:
      (. .. 猫.scm)

      plain-file:
      猫

      mixed-text-file:
      猫猫猫

      directory-union:
      (. .. 猫.scm)

Which is actually what the user would expect.

I also added missing arguments to the documentation.

* guix/gexp.scm (computed-file):  Set LANG to C.UTF-8 by default.
(compiled-modules): Try to `setlocale'.
(gexp->script), (gexp->file): New `locale' argument defaulting to C.UTF-8.
(text-file*): Set output port encoding to UTF-8.
* doc/guix.texi (G-Expressions)[computed-file]: Document the changes.  Use
@var.  Document #:guile.
[gexp->script]: Document #:locale.  Fix default value for #:target.
[gexp->file]: Document #:locale, #:system and #:target.

Change-Id: Ib323b51af88a588b780ff48ddd04db8be7c729fb
2025-01-12 16:19:02 +01:00
.mumi
build-aux
doc gexp: Improve support of Unicode characters. 2025-01-12 16:19:02 +01:00
etc
gnu gnu: ath9k-firmware: Remove CMAKE_SYSTEM_NAME override. 2025-01-12 16:19:02 +01:00
guix gexp: Improve support of Unicode characters. 2025-01-12 16:19:02 +01:00
m4
nix
po
scripts
tests
.dir-locals.el
.editorconfig
.gitattributes
.gitignore
.guix-authorizations
.guix-channel
.mailmap
.patman
AUTHORS
bootstrap
ChangeLog
CODE-OF-CONDUCT
config-daemon.ac
configure.ac
COPYING
gnu.scm
guix.scm
HACKING
Makefile.am
manifest.scm
NEWS
README
ROADMAP
THANKS
TODO

README

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

-*- mode: org -*-

[[https://www.gnu.org/software/guix/][GNU Guix]] (IPA: /ɡiːks/) is a purely functional package manager, and
associated free software distribution, for the [[https://www.gnu.org/gnu/gnu.html][GNU system]].  In addition
to standard package management features, Guix supports transactional
upgrades and roll-backs, unprivileged package management, per-user
profiles, and garbage collection.

It provides [[https://www.gnu.org/software/guile/][Guile]] Scheme APIs, including a high-level embedded
domain-specific languages (EDSLs) to describe how packages are to be
built and composed.

GNU Guix can be used on top of an already-installed GNU/Linux distribution, or
it can be used standalone (we call that “Guix System”).

Guix is based on the [[https://nixos.org/nix/][Nix]] package manager.


* Requirements

If you are building Guix from source, please see the manual for build
instructions and requirements, either by running:

  info -f doc/guix.info "Requirements"

or by checking the [[https://guix.gnu.org/manual/en/html_node/Requirements.html][web copy of the manual]].

* Installation

See the manual for the installation instructions, either by running

  info -f doc/guix.info "Installation"

or by checking the [[https://guix.gnu.org/manual/en/html_node/Installation.html][web copy of the manual]].

* Building from Git

For information on building Guix from a Git checkout, please see the relevant
section in the manual, either by running

  info -f doc/guix.info "Building from Git"

or by checking the [[https://guix.gnu.org/manual/en/html_node/Building-from-Git.html][web_copy of the manual]].

* How It Works

Guix does the high-level preparation of a /derivation/.  A derivation is
the promise of a build; it is stored as a text file under
=/gnu/store/xxx.drv=.  The (guix derivations) module provides the
`derivation' primitive, as well as higher-level wrappers such as
`build-expression->derivation'.

Guix does remote procedure calls (RPCs) to the build daemon (the =guix-daemon=
command), which in turn performs builds and accesses to the store on its
behalf.  The RPCs are implemented in the (guix store) module.

* Contact

GNU Guix is hosted at https://savannah.gnu.org/projects/guix/.

Please email <help-guix@gnu.org> for questions and <bug-guix@gnu.org> for bug
reports; email <gnu-system-discuss@gnu.org> for general issues regarding the
GNU system.

Join #guix on irc.libera.chat.

* Guix & Nix

GNU Guix is based on [[https://nixos.org/nix/][the Nix package manager]].  It implements the same
package deployment paradigm, and in fact it reuses some of its code.
Yet, different engineering decisions were made for Guix, as described
below.

Nix is really two things: a package build tool, implemented by a library
and daemon, and a special-purpose programming language.  GNU Guix relies
on the former, but uses Scheme as a replacement for the latter.

Using Scheme instead of a specific language allows us to get all the
features and tooling that come with Guile (compiler, debugger, REPL,
Unicode, libraries, etc.)  And it means that we have a general-purpose
language, on top of which we can have embedded domain-specific languages
(EDSLs), such as the one used to define packages.  This broadens what
can be done in package recipes themselves, and what can be done around them.

Technically, Guix makes remote procedure calls to the ‘nix-worker’
daemon to perform operations on the store.  At the lowest level, Nix
“derivations” represent promises of a build, stored in ‘.drv’ files in
the store.  Guix produces such derivations, which are then interpreted
by the daemon to perform the build.  Thus, Guix derivations can use
derivations produced by Nix (and vice versa).

With Nix and the [[https://nixos.org/nixpkgs][Nixpkgs]] distribution, package composition happens at
the Nix language level, but builders are usually written in Bash.
Conversely, Guix encourages the use of Scheme for both package
composition and builders.  Likewise, the core functionality of Nix is
written in C++ and Perl; Guix relies on some of the original C++ code,
but exposes all the API as Scheme.

* Related software

  - [[https://nixos.org][Nix, Nixpkgs, and NixOS]], functional package manager and associated
    software distribution, are the inspiration of Guix
  - [[https://www.gnu.org/software/stow/][GNU Stow]] builds around the idea of one directory per prefix, and a
    symlink tree to create user environments
  - [[https://www.pvv.ntnu.no/~arnej/store/storedoc_6.html][STORE]] shares the same idea
  - [[https://live.gnome.org/OSTree/][GNOME's OSTree]] allows bootable system images to be built from a
    specified set of packages
  - The [[https://www.gnu.org/s/gsrc/][GNU Source Release Collection]] (GSRC) is a user-land software
    distribution; unlike Guix, it relies on core tools available on the
    host system