117 Commits

Author SHA1 Message Date
Caleb Connolly
b54044a605 hookscripts: glob hook directories 2023-04-18 17:53:37 +01:00
Caleb Connolly
f0544999db filelist/hookfiles: encapsulate dir searching out of slurpFiles
Move the directory searching / globbing code out of slurpFiles and into
the filelist module so it can be used elsewhere.
2023-04-07 02:56:38 +01:00
Clayton Craft
8fac3004a6 archive: fix up documentation for AddItems 2023-03-19 23:14:18 -07:00
Clayton Craft
a15a3ad781 filelist/modules: don't print "skipping..." when dir not found
I think this was still causing some confusion, since it *might* look
like a failure when in reality it's not. I think it's important that
mkinitfs prints when it is adding something, and doesn't print when it
is *not* adding something, so that it should be clear if something
expected is missing and when something unexpected is included... without
having to sort out which is which every time the output is read.
2023-03-19 23:06:09 -07:00
Clayton Craft
1e8580a0a1 archive: New() can't fail, so don't return an error type
It could fail if the system can't allocate memory or something
undeterministic like that, but in that case it's best to just let the
runtime panic.
2023-03-19 22:05:55 -07:00
Clayton Craft
e6ee43826d doc/mkinitfs: add section on boot-deploy 2023-03-19 16:36:11 -07:00
Clayton Craft
7bdd68800d doc/mkinitfs.1: add design goals 2023-03-19 16:35:51 -07:00
Clayton Craft
80098d29c6 misc:TimeFunc: reduce printed time precision
Now:
        10:57:41.737665 initramfs-extra completed in: 0.33s
        ...
        10:57:42.008587 boot-deploy completed in: 0.27s
        10:57:42.012973 mkinitfs completed in: 0.90s

Times is just truncated, not rounding, since it's simpler (no dependency
on the math module), and I'm not sure if anyone cares for what this
function prints. If there is a desire to return to higher precision
later, it could be enabled by a new flag.

fixes https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/25
2023-03-13 11:03:48 -07:00
Clayton Craft
67f1839ddc filelist/modules: fix order of struct items to reduce memory
Another one found by fieldalignment:
        modules.go:18:14: struct with 32 pointer bytes could be 24

Probably not going to matter much... but let's just get rid of the
warning.
2023-03-12 20:39:01 -07:00
Clayton Craft
baf76ed614 archive: fix order of struct items to reduce memory usage
Found by fieldalignment:
        archive.go:46:14: struct with 88 pointer bytes could be 56
        archive.go:66:18: struct with 24 pointer bytes could be 16

The first one probably doesn't matter that much, there's only like 2 of
those objects that are instantiated at runtime. However, there are many
ArchiveItems (hundreds or more depending on the archives compositions)
2023-03-12 20:37:10 -07:00
Clayton Craft
27e271b904 filelist/modules: print a message when including modules from deviceinfo
fixes https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/24
2023-03-12 20:36:29 -07:00
Clayton Craft
1ac85b12fe filelist/modules: print search dir before searching dir
This will allow printing status for deviceinfo modules earlier in the
function, in a way that makes more sense.
2023-03-12 20:36:24 -07:00
Clayton Craft
f7f42bc2d4 filelist/modules: handle errors from filepath.Walk 2023-03-12 20:18:51 -07:00
Clayton Craft
c62a1f9ddb filelist/modules: remove outdated reference in error message 2023-03-12 20:05:55 -07:00
Clayton Craft
c9de619f98 filelist/hookscripts: use the correct path for scripts 2023-03-10 23:23:53 -08:00
Clayton Craft
a519769979 filelist/*: use "searching" instead of "including" in top-level msg
When this is printed, it's about to search the given path for stuff to
slurp up, but it hasn't actually included anything yet. Using
"Including" here was kinda confusing, so this changes it to use
"Searching for." Hopefully that's better!
2023-03-10 22:33:32 -08:00
Clayton Craft
128a48dd24 cmd/mkinitfs: re-add support for setting initramfs-extra compression 2023-03-10 22:22:51 -08:00
Clayton Craft
499136e83a cmd/mkinitfs: move printing compression info to generateArchive 2023-03-10 22:22:51 -08:00
Clayton Craft
78f8fa32fb deviceinfo: add initfs_extra_compression
For configuring the archive compression parameters for the
initramfs-extra archive.
2023-03-10 22:22:50 -08:00
Clayton Craft
d03257981f bootdeploy: add context to kernel copy fd close error (MR 33) 2023-03-06 22:20:11 -08:00
Clayton Craft
307fb1889f bootdeploy: use original kernel filename when calling boot-deploy (MR 33)
fixes #21
2023-03-06 22:20:07 -08:00
Clayton Craft
fa3d3268d7 bootdeploy: catch any errors when closing kernel file copy fd 2023-03-05 23:56:58 -08:00
Clayton Craft
8b67848d5c filelist/modules: print search path for modules
By doing so, it adds more useful context to this:

         - Including kernel modules
         -- Unable to find dir, skipping...
         - Including kernel modules
         -- Unable to find dir, skipping...
2023-03-05 23:54:15 -08:00
Clayton Craft
31ab72edbc cmd/mkinitfs: configure log to print milliseconds
Useful for timing each step of generating the initramfs without having
to add a bunch of calls to the "time func" function
2023-03-01 22:57:53 -08:00
Clayton Craft
bd239c0365 ci: build vendor tarball and create new release when new tag is pushed
This adds two new CI jobs that run when a new tag is pushed to this
repo: one to tar up the vendored dependencies and another one to push
the tarball as an asset in a new gitlab "release" for the tag.

fixes: #5
2023-03-01 12:56:21 -08:00
Clayton Craft
a4c3b9ff96 Makefile: add target to build vendored dependencies 2023-03-01 12:56:21 -08:00
Clayton Craft
8f505ffdc8 go.mod: set min version to 1.20
Fixes some build issues when using vendored dependencies.
2023-03-01 12:56:21 -08:00
Clayton Craft
fb00e9e94b cmd/mkinitfs: don't compress initramfs-extra (MR 25)
There is little(?) reason to compress the -extra archive, but some
there are some good reasons why it should be left uncompressed:

        1) decompression increases boot time
        2) selecting exotic formats (e.g. lzma or zstd) can actually
           increase the initramfs size since you now have to include
           support for those archive formats

In the future this could be made configurable...
2023-02-26 12:18:27 -08:00
Clayton Craft
7c2377d0c8 archive: add "none" compression format (MR 25) 2023-02-26 12:18:27 -08:00
Clayton Craft
f24d0139c9 archive.writeCompressed: bubble up error when closing archive file (MR 25)
This also prevents a double Close() if the compressor is ever set to 'fd'
2023-02-26 12:17:58 -08:00
Clayton Craft
5e2f975bd3 cmd/mkinitfs: print info about compression format and level used (MR 25) 2023-02-24 09:21:24 -08:00
Clayton Craft
786e09d855 archive: support using lzma (MR 25)
fixes: https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/2
2023-02-24 09:21:24 -08:00
Clayton Craft
ba1e1a77db archive: Use compression level when generating archive (MR 25) 2023-02-24 09:21:24 -08:00
Clayton Craft
fd11f4a627 archive.New: accept compression level (MR 25)
I went with a simpler implementation that uses Go compression packages
to do the work. The downside of this is that the compression Level is a
bit weird to set, since most libraries discourage setting the numeric
compression level directly.

This is configured by setting `deviceinfo_initfs_compression`, the value
it expects is a string in the form: `FORMAT[:LEVEL]`, where `[:LEVEL]`
is optional. Actually setting the variable at all is optional... if
nothing is specified, or it can't parse the format/level from the string
value, it defaults to using gzip with the "default" level for the
package (which tries to mirror gzip's default, or something).

The level can be one of `default`, `fast`, `best`.

To configure gzip with the fastest compression (so, bigger size): deviceinfo_initfs_compression="gzip:fast"`

To configure zstd with the most compression: `deviceinfo_initfs_compression="zstd:best"`

To configure zstd with default compression: `deviceinfo_initfs_compression="zstd"` (or `deviceinfo_initfs_compression="zstd:default"`)

In this case, `gzip:default` is assumed: deviceinfo_initfs_compression="bananas:mmmm"`
2023-02-24 09:21:23 -08:00
Clayton Craft
322d6bb754 archive: add ExtractFormatLevel function (MR 25)
Extracts the format and level from a string in the form "format:level"
2023-02-24 09:21:23 -08:00
Clayton Craft
1f4d8737e8 archive: add CompressLevel type and consts (MR 25)
There's really not a great way to map individual levels to each
compression library, so this just adds a new type that will invoke the
three relevant levels for each library used. This could be improved in
the future.
2023-02-24 09:21:23 -08:00
Clayton Craft
52fc741ba8 archive: support using zstd (MR 25)
This external module was chosen because it's a native Go implementation
of zstd, and not a wrapper around some external utility or some CGO
thing.
2023-02-24 09:21:23 -08:00
Clayton Craft
31b7eb34ee archive: allow selecting different formats when writing compressed file (MR 25) 2023-02-24 09:21:23 -08:00
Clayton Craft
4e97990804 archive: accept compression format parameter during instantiation (MR 25) 2023-02-24 09:21:23 -08:00
Clayton Craft
c01b48ad25 archive: add CompressFormat type with initial constants (MR 25) 2023-02-24 09:21:22 -08:00
Clayton Craft
6aec4d564c archive: remove pgzip, use gzip from go std lib (MR 25)
This replaces the parallel gzip with the boring gzip from Go's standard
lib. The main motivations for doing this are:

1) Reduce runtime memory requirements

2) shed some external dependencies

There's obviously a trade-off with compression speed/time (as seen
below), but I feel like it's a worthwhile trade-off.

Note that there's likely very little impact to boot performance wrt
extracting these archives, the compression levels are similar.

Measured on a Shift 6mq, which is a very fast phone...

** compress/gzip:

User time (seconds): 1.81
System time (seconds): 0.38
Percent of CPU this job got: 104%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.09
Maximum resident set size (kbytes): 62024

-rw-r--r-- 1 clayton clayton 6.1M Sep 20 17:20 initramfs
-rw-r--r-- 1 clayton clayton 2.5M Sep 20 17:20 initramfs-extra

** pgzip:

User time (seconds): 1.19
System time (seconds): 0.48
Percent of CPU this job got: 159%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.05
Maximum resident set size (kbytes): 109952

-rw-r--r-- 1 clayton clayton 6.8M Sep 20 17:20 initramfs
-rw-r--r-- 1 clayton clayton 2.8M Sep 20 17:20 initramfs-extra

inspired by: https://gitlab.com/postmarketOS/pmaports/-/issues/1704
2023-02-24 09:21:22 -08:00
Clayton Craft
6eb01e91e6 cmd/mkinitfs: actually read modules from /etc/mkinitfs/modules (MR 32)
Missed earlier. Note that the list of modules from deviceinfo isn't
passed again...
2023-02-24 09:06:40 -08:00
Clayton Craft
790cf47060 cmd/mkinitfs: allow installing modules in the initramfs-extra archive (MR 32)
fixes: https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/20
2023-02-24 09:06:29 -08:00
Clayton Craft
4074eada55 modules: remove TODOs
The builtin one was moved to a new issue:
https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/18

The "more extensions" one was not moved to a new issue, it should be
obvious if we do need to support more extensions.
2023-02-21 12:41:18 -08:00
Clayton Craft
a7c4fe83ce archive: remove TODO about adding debug mode
moved to an issue: https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/17
2023-02-21 12:41:18 -08:00
Clayton Craft
06f86aadc9 misc.Exists: bubble up any unexpected errors
Fixes https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/issues/6
2023-02-21 12:41:18 -08:00
Clayton Craft
d87a33a751 bootdeploy: remove dependency on misc.Exists
Checking for the app is unnecessary, since the cmd.Run later will fail
if it doesn't exist. This allows dropping the dependency on misc.Exists.

There's also no reason to print that the command failed, just return the
error.
2023-02-21 12:41:18 -08:00
Clayton Craft
d1e150242d cmd/mkinitfs: fix boot-deploy error message
naming the app is more useful than some internal function/object
2023-02-21 12:41:18 -08:00
Clayton Craft
5968622f60 cmd/mkinitfs: simplify the "deviceinfo not found" message
The old message is too specific to pmOS.
2023-02-21 12:41:18 -08:00
Clayton Craft
0179a0ca5c misc: remove functions now in osutils, update references in project 2023-02-21 12:41:17 -08:00
Clayton Craft
33c61b3c94 internal/osutil: new package
This is a collection of things from internal/misc that should have been
separated from misc.
2023-02-21 12:41:17 -08:00
Clayton Craft
e4fb6cef70 README: update description and usage sections to match recent changes 2023-02-21 12:41:17 -08:00
Clayton Craft
4ae678d8ce cmd/mkinitfs: add -no-bootdeploy option to disable boot-deploy 2023-02-21 12:41:17 -08:00
Clayton Craft
71c2a87d56 cmd/mkinitfs: fix running deferred functions from main on error
This new style is a little more verbose (having to manually set return
code on error..), but at least it offers a chance to improve the
printing of errors a little more.
2023-02-21 00:47:05 -08:00
Clayton Craft
9bb326be91 bootdeploy: describe copyUbootFiles 2023-02-21 00:47:05 -08:00
Clayton Craft
0545d68b1d bootdeploy: return errors from writing copy 2023-02-21 00:47:05 -08:00
Clayton Craft
c6e79551f4 cmd/mkinitfs: show time spent generating each archive 2023-02-21 00:47:04 -08:00
Clayton Craft
a9f4281fbd cmd/mkinitfs: show time spent running boot-deploy 2023-02-21 00:47:04 -08:00
Clayton Craft
bb50041257 cmd/mkinitfs: move printing "== Generating.." message to generateArchive 2023-02-21 00:47:04 -08:00
Clayton Craft
09c897e737 cmd/mkinitfs: use a BootDeploy type for running boot-deploy 2023-02-21 00:47:04 -08:00
Clayton Craft
a8bb10ce9c bootdeploy: copy implementation from cmd/mkinitfs 2023-02-21 00:47:04 -08:00
Clayton Craft
5e65ace958 internal/bootdeploy: add new package
As a stub, but will be implemented soon...
2023-02-21 00:47:04 -08:00
Clayton Craft
cbcd4408e3 doc/mkinitfs.1: Fill in DEVICEINFO section 2023-02-21 00:47:03 -08:00
Clayton Craft
ad560591e1 deviceinfo: parse only required variables
This greatly reduces the chance accidentally adding dependencies to the
other (currently unused) variables later on. Getting away from depending
on deviceinfo has a lot of benefits, but mainly it helps offload
device-specific boot configuration to boot-deploy. Handling those
complexities in a shell script is often nicer.

Also, reducing the need to handle variables that contain lists means
that this app doesn't have to worry about how to merge/handle multiple
versions of those. That might be useful later if mkinitfs has to read
deviceinfo config from multiple deviceinfo files.

For example, trying to figure out how to merge these two things is...
ehhh...
        a_modules_initfs="abc bar banana bazz"
        b_modules_initfs="foo bar bazz bar2 guava"
2023-02-21 00:47:03 -08:00
Clayton Craft
89f1e067da cmd/mkinitfs: add --version option 2023-02-21 00:47:03 -08:00
Clayton Craft
4259478755 doc/mkinitfs.1: add manpage 2023-02-21 00:47:03 -08:00
Clayton Craft
347668caa3 cmd/mkinitfs: use filelist/hookdirs when creating initramfs 2023-02-21 00:47:03 -08:00
Clayton Craft
b0e28b4215 filelist/hookdirs: add new FileLister implementation for creating dirs 2023-02-21 00:47:03 -08:00
Clayton Craft
c1d96f699c filelist/hookscripts: don't fail if directory not found 2023-02-21 00:47:02 -08:00
Clayton Craft
25c3c03e24 filelist/osksdl: fix crash when osk-sdl isn't found
Should have returned an empty FileList in this case
2023-02-21 00:47:02 -08:00
Clayton Craft
07c8c711c7 filelist/hookfiles: don't error out if dir is not found
Just print a message that it wasn't found, instead of sending an error
that may stop the entire app.
2023-02-21 00:47:02 -08:00
Clayton Craft
e772fe0c87 filelist/hookfiles: support specifying src:dest in .files 2023-02-21 00:47:02 -08:00
Clayton Craft
6f05222018 filelist/modules: add support for directories in .modules files
This is kinda janky, passing both a list of modules and a directory path
is just because we support modules in deviceinfo and in directory file
lists... Maybe one day we can move away from having modules in the
deviceinfo...
2023-02-21 00:45:58 -08:00
Clayton Craft
c23af8b541 cmd/mkinitfs: simplify archive creation by using filelisters/features
I'm going to rename "filelist" to "feature" soon...
2023-02-21 00:45:58 -08:00
Clayton Craft
bd09de9232 archive: accept a FileLister in the AddItems method 2023-02-21 00:45:58 -08:00
Clayton Craft
22692e48d2 filelist/initramfs: add new type for slurping up file listers 2023-02-21 00:45:58 -08:00
Clayton Craft
6c2f7b972b filelist/*: implement FileLister in existing types 2023-02-21 00:45:57 -08:00
Clayton Craft
e5002f5750 filelist/FileList: add Import method 2023-02-21 00:45:16 -08:00
Clayton Craft
662f559286 filelist: Add FileList type
This adds a new type, FileList, to manage another new type (File). File
contains the mapping of source --> dest for a file or directory.
2023-02-21 00:45:16 -08:00
Clayton Craft
a4be663e13 filelist/*.List(): print to stdout when starting to generate FileList 2023-02-21 00:45:16 -08:00
Clayton Craft
14873015c0 cmd/mkinitfs: move getInitfsFiles into generateInitfs
Prep for simplifying things a bit...
2023-02-21 00:45:16 -08:00
Clayton Craft
6fdc8937b5 internal/archive: make archiveItem.add private 2023-02-21 00:45:16 -08:00
Clayton Craft
fb52066d8f filelist/modules: remove postmarketos- from config dir name 2023-02-21 00:45:15 -08:00
Clayton Craft
b7f520cba4 filelist/modules: drop "required modules"
Breaking change. These should be provided in /*/modules/* lists
2023-02-21 00:45:15 -08:00
Clayton Craft
31bf38f663 cmd/mkinitfs: don't hardcode splash images
No longer supported/used on pmOS since pbsplash was merged.
2023-02-21 00:45:15 -08:00
Clayton Craft
71d8131bb0 cmd/mkinitfs: don't hardcode required directories
Another breaking change, these directories should be specified via
hook files
2023-02-21 00:45:15 -08:00
Clayton Craft
8b99b5f45b cmd/mkinitfs: don't hardcode binaries/files for including in archives
Obvious breaking change... These files should be provided by "hook file"
lists instead.
2023-02-21 00:45:15 -08:00
Clayton Craft
e8854ff88d cmd/mkinitfs: drop getHookFiles 2023-02-21 00:45:15 -08:00
Clayton Craft
1eb35cf8ef cmd/mkinitfs: drop getHookScripts 2023-02-21 00:45:14 -08:00
Clayton Craft
696633629a cmd/mkinitfs: drop getFiles + friends
moved to misc package
2023-02-21 00:45:14 -08:00
Clayton Craft
d9b68843a3 cmd/mkinitfs: use misc.GetFiles 2023-02-21 00:45:14 -08:00
Clayton Craft
93005527e0 cmd/mkinitfs: drop getKernelVersion and getKernelReleaseFile 2023-02-21 00:45:14 -08:00
Clayton Craft
1c5f16762f cmd/mkinitfs: use misc.GetKernelVersion 2023-02-21 00:45:14 -08:00
Clayton Craft
af97d4654f cmd/mkinitfs: use filelist/hookfiles 2023-02-21 00:45:14 -08:00
Clayton Craft
b25c9bd390 filelist/hookfiles: add new implementation 2023-02-21 00:45:13 -08:00
Clayton Craft
1a0d00e39f cmd/mkinitfs: use filelist/hookscripts 2023-02-21 00:45:13 -08:00
Clayton Craft
af3c47c784 filelist/hookscripts: add new implementation 2023-02-21 00:45:13 -08:00
Clayton Craft
e7bbd1cadf cmd/mkinitfs: use filelist/modules 2023-02-21 00:45:13 -08:00
Clayton Craft
1531d7e790 filelist/modules: add new implementation 2023-02-21 00:45:13 -08:00
Clayton Craft
6d77b7a2d1 cmd/mkinitfs: drop osksdl-related stuff 2023-02-21 00:45:13 -08:00
Clayton Craft
2dd83da480 cmd/mkinitfs: use filelist/osksdl 2023-02-21 00:45:12 -08:00
Clayton Craft
e00e5faf6e filelist/osksdl: just use fmt.Errorf to create an error 2023-02-21 00:45:12 -08:00
Clayton Craft
5e07b63084 filelist/osksdl: add new implementation 2023-02-21 00:45:12 -08:00
Clayton Craft
95582ee034 misc: add GetFiles and supporting functions 2023-02-18 11:22:26 -08:00
Clayton Craft
94584050ee cmd/mkinitfs: use misc.Exists, drop exists() 2023-02-18 11:22:25 -08:00
Clayton Craft
e0977b4ac1 misc: add Exists() 2023-02-18 11:22:25 -08:00
Clayton Craft
4176a8a661 misc: add GetKernelVersion()
Allows re-using across different packages
2023-02-18 11:22:25 -08:00
Clayton Craft
73fd85f68c filelist: add FileLister interface 2023-02-18 11:22:25 -08:00
Clayton Craft
7e80107bbe misc: add TimeFunc()
This allows it to be re-used easily when things are broken up into
more packages later.
2023-02-17 14:21:14 -08:00
Clayton Craft
f714f110a1 cmd/mkinitfs: rename binary and move
This moves the main package to cmd/mkinitfs, and configures the compiled
binary to be named 'mkinitfs'. calling the full name
'postmarketos-mkinitfs' was unlikely to be used by anyone...

This move makes the project source layout more consistent with other Go
projects, and allows for adding more cmd/* things with their own main
packages later if we want
2023-02-17 14:21:14 -08:00
Clayton Craft
690d008643 internal/{archive,misc}: move from pkgs
These should be internal-only to this project
2023-02-09 13:34:06 -08:00
Clayton Craft
731a805a9e getBinaryDeps: properly handle circular lib dependencies
This moves the recursive bit outside of getBinaryDeps and properly
handles circular dependencies.
2023-02-09 12:42:06 -08:00
Clayton Craft
b90624d7dd getBinaryDeps: move recursive bit to new function 2023-02-09 12:41:34 -08:00
Oliver Smith
2a75cf9b4e getInitfsExtraFiles: add scripts from hooks-extra (MR 27)
Put scripts from /etc/postmarketos-mkinitfs/hooks-extra into the extra
initramfs instead of the regular one, similar to how it is possible with
files listed in /etc/postmarketos-mkinitfs/files-extra.

This way we will be able to launch hooks not only very early in the
initramfs as it's currently the case. But also later on after the
initramfs-extra was extracted, and more files are available. ondev2 will
make use of this feature.
2023-01-17 08:12:03 +01:00
Oliver Smith
d52cc16c88 getBinaryDeps: search in /usr/lib/expect* (MR 28)
Instead of only searching for shared libraries in /usr/lib and /lib,
also search in /usr/lib/expect* (currently the expect binary links
against /usr/lib/expect5.45.4/libexpect5.45.4.so).

I've also considered searching /usr/lib recursively, but that would be a
major performance hit.

Expect gets added to the initramfs-extra in a script that runs the
ondev2 testsuite inside qemu.
2023-01-16 22:55:36 -08:00
Oliver Smith
112b572dc2 getFile: print exact error from os.Stat (MR 28)
Pass the exact error message down and make formatting of the message
consistent.
2023-01-16 22:55:30 -08:00
Oliver Smith
0c0a85f3bb getHookFiles: print exact error from getFiles (MR 28)
Instead of assuming that the error is "unable to find file", print the
actual error from getFiles. I just had a situation where the file
exists, but a dependency couldn't be found.

Before:
  generateInitfsExtra: getHookFiles: unable to find file "/usr/bin/expect" required by "/etc/postmarketos-mkinitfs/files-extra/ondev2-test.files"

After:
  generateInitfsExtra: getHookFiles: unable to add file "/usr/bin/expect" required by "/etc/postmarketos-mkinitfs/files-extra/ondev2-test.files": getBinaryDeps: unable to locate dependency for "/usr/bin/expect": libexpect5.45.4.so
2023-01-16 22:55:18 -08:00
26 changed files with 1729 additions and 990 deletions

6
.gitignore vendored
View File

@@ -1 +1,5 @@
/postmarketos-mkinitfs
/*.1
/*.tar.gz
/*.sha512
/mkinitfs
/vendor

View File

@@ -5,10 +5,13 @@ image: alpine:edge
variables:
GOFLAGS: "-buildvcs=false"
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/mkinitfs-vendor-${CI_COMMIT_TAG}/${CI_COMMIT_TAG}"
stages:
- lint
- build
- vendor
- release
# defaults for "only"
# We need to run the CI jobs in a "merge request specific context", if CI is
@@ -28,9 +31,33 @@ build:
stage: build
<<: *only-default
before_script:
- apk -q add go staticcheck make
- apk -q add go staticcheck make scdoc
script:
- make test
- make
artifacts:
expire_in: 1 week
vendor:
stage: vendor
image: alpine:latest
only:
- tags
before_script:
- apk -q add curl go make
script:
- |
make VERSION="${CI_COMMIT_TAG}" vendor
curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file "mkinitfs-vendor-${CI_COMMIT_TAG}.tar.gz" "${PACKAGE_REGISTRY_URL}/"
curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file "mkinitfs-vendor-${CI_COMMIT_TAG}.tar.gz.sha512" "${PACKAGE_REGISTRY_URL}/"
release:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
only:
- tags
script:
- |
release-cli create --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG \
--assets-link "{\"name\":\"mkinitfs-vendor-${CI_COMMIT_TAG}.tar.gz\",\"url\":\"${PACKAGE_REGISTRY_URL}/mkinitfs-vendor-${CI_COMMIT_TAG}.tar.gz\"}" \
--assets-link "{\"name\":\"mkinitfs-vendor-${CI_COMMIT_TAG}.tar.gz.sha512\",\"url\":\"${PACKAGE_REGISTRY_URL}/mkinitfs-vendor-${CI_COMMIT_TAG}.tar.gz.sha512\"}"

View File

@@ -1,22 +1,34 @@
.POSIX:
.SUFFIXES:
.SUFFIXES: .1 .1.scd
VERSION?=$(shell git describe --tags --dirty 2>/dev/null || echo 0.0.0)
VPATH=doc
VENDORED="mkinitfs-vendor-$(VERSION)"
PREFIX?=/usr/local
BINDIR?=$(PREFIX)/sbin
MANDIR?=$(PREFIX)/share/man
SHAREDIR?=$(PREFIX)/share
GO?=go
GOFLAGS?=
LDFLAGS+=-s -w
LDFLAGS+=-s -w -X main.Version=$(VERSION)
RM?=rm -f
GOTEST=go test -count=1 -race
GOSRC!=find * -name '*.go'
GOSRC+=go.mod go.sum
all: postmarketos-mkinitfs
DOCS := \
mkinitfs.1
postmarketos-mkinitfs: $(GOSRC)
$(GO) build $(GOFLAGS) -ldflags "$(LDFLAGS)" -o postmarketos-mkinitfs
all: mkinitfs $(DOCS)
mkinitfs: $(GOSRC)
$(GO) build $(GOFLAGS) -ldflags "$(LDFLAGS)" -o mkinitfs ./cmd/mkinitfs
.1.scd.1:
scdoc < $< > $@
doc: $(DOCS)
.PHONY: fmt
fmt:
@@ -33,22 +45,30 @@ test:
@$(GOTEST) ./...
clean:
$(RM) postmarketos-mkinitfs
$(RM) mkinitfs $(DOCS)
$(RM) $(VENDORED)*
install: $(DOCS) postmarketos-mkinitfs
install -Dm755 postmarketos-mkinitfs -t $(DESTDIR)$(BINDIR)/
ln -sf postmarketos-mkinitfs $(DESTDIR)$(BINDIR)/mkinitfs
install: $(DOCS) mkinitfs
install -Dm755 mkinitfs -t $(DESTDIR)$(BINDIR)/
install -Dm644 mkinitfs.1 -t $(DESTDIR)$(MANDIR)/man1/
.PHONY: checkinstall
checkinstall:
test -e $(DESTDIR)$(BINDIR)/postmarketos-mkinitfs
test -L $(DESTDIR)$(BINDIR)/mkinitfs
test -e $(DESTDIR)$(BINDIR)/mkinitfs
test -e $(DESTDIR)$(MANDIR)/man1/mkinitfs.1
RMDIR_IF_EMPTY:=sh -c '! [ -d $$0 ] || ls -1qA $$0 | grep -q . || rmdir $$0'
vendor:
go mod vendor
tar czf $(VENDORED).tar.gz vendor/
sha512sum $(VENDORED).tar.gz > $(VENDORED).tar.gz.sha512
$(RM) -rf vendor
uninstall:
$(RM) $(DESTDIR)$(BINDIR)/postmarketos-mkinitfs
$(RM) $(DESTDIR)$(BINDIR)/mkinitfs
${RMDIR_IF_EMPTY} $(DESTDIR)$(BINDIR)
$(RM) $(DESTDIR)$(MANDIR)/man1/mkinitfs.1
$(RMDIR_IF_EMPTY) $(DESTDIR)$(MANDIR)/man1
.PHONY: all clean install uninstall test
.PHONY: all clean install uninstall test vendor

View File

@@ -1,5 +1,8 @@
`postmarketos-mkinitfs` is a tool for generating an initramfs (and installing
it) on postmarketOS.
`mkinitfs` is a tool for generating an initramfs. It was originally designed
for postmarketOS, but a long term design goal is to be as distro-agnostic as
possible. It's capable of generating a split initramfs, in the style used by
postmarketOS, and supports running `boot-deploy` to install/finalize boot files
on a device.
## Building
@@ -34,12 +37,12 @@ $ make test
## Usage
The application uses configuration from `/etc/deviceinfo`, and does not support
any other options at runtime. It can be run simply by executing:
The tool can be run with no options:
```
$ postmarketos-mkinitfs
# mkinitfs
```
For historical reasons, a symlink from `mkinitfs` to `postmarketos-mkinitfs` is
also installed by the makefile's `install` target.
Configuration is done through a series of flat text files that list directories
and files, and by placing scripts in specific directories. See `man 1 mkinitfs`
for more information.

168
cmd/mkinitfs/main.go Normal file
View File

@@ -0,0 +1,168 @@
// Copyright 2022 Clayton Craft <clayton@craftyguy.net>
// SPDX-License-Identifier: GPL-3.0-or-later
package main
import (
"flag"
"fmt"
"log"
"os"
"path/filepath"
"strings"
"time"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/archive"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/bootdeploy"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist/hookdirs"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist/hookfiles"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist/hookscripts"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist/initramfs"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist/modules"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist/osksdl"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/misc"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/osutil"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/pkgs/deviceinfo"
)
// set at build time
var Version string
func main() {
retCode := 0
defer func() { os.Exit(retCode) }()
outDir := flag.String("d", "/boot", "Directory to output initfs(-extra) and other boot files")
var showVersion bool
flag.BoolVar(&showVersion, "version", false, "Print version and quit.")
var disableBootDeploy bool
flag.BoolVar(&disableBootDeploy, "no-bootdeploy", false, "Disable running 'boot-deploy' after generating archives.")
flag.Parse()
if showVersion {
fmt.Printf("%s - %s\n", filepath.Base(os.Args[0]), Version)
return
}
log.Default().SetFlags(log.Lmicroseconds)
deviceinfoFile := "/etc/deviceinfo"
if exists, err := misc.Exists(deviceinfoFile); !exists {
log.Printf("NOTE: %q not found, this file is required by mkinitfs.\n", deviceinfoFile)
return
} else if err != nil {
retCode = 1
log.Printf("received unexpected error when getting status for %q: %s", deviceinfoFile, err)
}
devinfo, err := deviceinfo.ReadDeviceinfo(deviceinfoFile)
if err != nil {
log.Println(err)
retCode = 1
return
}
defer misc.TimeFunc(time.Now(), "mkinitfs")
kernVer, err := osutil.GetKernelVersion()
if err != nil {
log.Println(err)
retCode = 1
return
}
// temporary working dir
workDir, err := os.MkdirTemp("", "mkinitfs")
if err != nil {
log.Println(err)
log.Println("unable to create temporary work directory")
retCode = 1
return
}
defer func() {
e := os.RemoveAll(workDir)
if e != nil && err == nil {
err = e
retCode = 1
}
}()
log.Print("Generating for kernel version: ", kernVer)
log.Print("Output directory: ", *outDir)
// deviceinfo.InitfsCompression needs a little more post-processing
compressionFormat, compressionLevel := archive.ExtractFormatLevel(devinfo.InitfsCompression)
if err := generateArchive("initramfs", compressionFormat, compressionLevel, workDir, []filelist.FileLister{
hookdirs.New("/usr/share/mkinitfs/dirs"),
hookdirs.New("/etc/mkinitfs/dirs"),
hookfiles.New("/usr/share/mkinitfs/files"),
hookfiles.New("/etc/mkinitfs/files"),
hookscripts.New("/usr/share/mkinitfs/hooks", "/hooks"),
hookscripts.New("/etc/mkinitfs/hooks", "/hooks"),
modules.New(strings.Fields(devinfo.ModulesInitfs), "/usr/share/mkinitfs/modules"),
modules.New([]string{}, "/etc/mkinitfs/modules"),
}); err != nil {
log.Println(err)
log.Println("failed to generate: ", "initramfs")
retCode = 1
return
}
// deviceinfo.InitfsExtraCompression needs a little more post-processing
compressionFormat, compressionLevel = archive.ExtractFormatLevel(devinfo.InitfsExtraCompression)
if err := generateArchive("initramfs-extra", compressionFormat, compressionLevel, workDir, []filelist.FileLister{
hookfiles.New("/usr/share/mkinitfs/files-extra"),
hookfiles.New("/etc/mkinitfs/files-extra"),
hookscripts.New("/usr/share/mkinitfs/hooks-extra", "/hooks-extra"),
hookscripts.New("/etc/mkinitfs/hooks-extra", "/hooks-extra"),
modules.New([]string{}, "/usr/share/mkinitfs/modules-extra"),
modules.New([]string{}, "/etc/mkinitfs/modules-extra"),
osksdl.New(devinfo.MesaDriver),
}); err != nil {
log.Println(err)
log.Println("failed to generate: ", "initramfs-extra")
retCode = 1
return
}
// Final processing of initramfs / kernel is done by boot-deploy
if !disableBootDeploy {
if err := bootDeploy(workDir, *outDir, devinfo.UbootBoardname); err != nil {
log.Println(err)
log.Println("boot-deploy failed")
retCode = 1
return
}
}
}
func bootDeploy(workDir, outDir, ubootBoardname string) error {
log.Print("== Using boot-deploy to finalize/install files ==")
defer misc.TimeFunc(time.Now(), "boot-deploy")
bd := bootdeploy.New(workDir, outDir, ubootBoardname)
return bd.Run()
}
func generateArchive(name string, format archive.CompressFormat, level archive.CompressLevel, path string, features []filelist.FileLister) error {
log.Printf("== Generating %s ==\n", name)
log.Printf("- Using compression format %s with level %q\n", format, level)
defer misc.TimeFunc(time.Now(), name)
a := archive.New(format, level)
fs := initramfs.New(features)
if err := a.AddItems(fs); err != nil {
return err
}
log.Println("- Writing and verifying archive: ", name)
if err := a.Write(filepath.Join(path, name), os.FileMode(0644)); err != nil {
return err
}
return nil
}

155
doc/mkinitfs.1.scd Normal file
View File

@@ -0,0 +1,155 @@
mkinitfs(1) "mkinitfs"
# NAME
mkinitfs
# DESCRIPTION
mkinitfs is a simple, generic tool for generating an initramfs, primarily
developed for use in postmarketOS
# CONCEPTS
mkinitfs is designed to generate two archives, "initramfs" and
"initramfs-extra", however it's possible to configure mkinitfs to run without
generating an initramfs-extra archive. mkinitfs is primarily configured through
the placement of files in specific directories detailed below in the
*DIRECTORIES* section. *deviceinfo* files are also used to provide other
configuration options to mkinitfs, these are covered under the *DEVICEINFO*
section below.
mkinitfs does not provide an init script, or any boot-time logic, it's purpose
is purely to generate the archive(s). mkinitfs does call *boot-deploy* after
creating the archive(s), in order to install/deploy them and any other relevant
boot-related items onto the system.
Design goals of this project are:
- Support as many distros as possible
- Simplify configuration, while still giving multiple opportunities to set or override defaults
- Execute an external app to do any boot install/setup finalization
- One such app is here: https://gitlab.com/postmarketOS/boot-deploy
- But implementation can be anything, see the section on *BOOT-DEPLOY*
for more info
# DEVICEINFO
The canonical deviceinfo "specification" is at
https://wiki.postmarketos.org/wiki/Deviceinfo_reference
mkinitfs reads deviceinfo values from */etc/deviceinfo*. The following variables
are *required* by mkinitfs:
- deviceinfo_initfs_compression
- deviceinfo_initfs_extra_compression
- deviceinfo_mesa_driver
- deviceinfo_modules_initfs
- deviceinfo_uboot_boardname
It is a design goal to keep the number of required variables from deviceinfo to
a bare minimum, and to require only variables that don't hold lists of things.
*NOTE*: When deviceinfo_initfs_extra_compression is set, make sure that the
necessary tools to extract the configured archive format are in the initramfs
archive.
# DIRECTORIES
The following directories are used by mkinitfs to generate the initramfs and
initramfs-extra archives. Directories that end in *-extra* indicate directories
that are used for constructing the initramfs-extra archive, while those without
it are for constructing the initramfs archive.
Configuration under */usr/share/mkinitfs* is intended to be managed by
distributions, while configuration under */etc/mkinitfs* is for users to
create/manage. mkinitfs reads configuration from */usr/share/mkinitfs* first, and then from */etc/mkinitfs*.
## /usr/share/mkinitfs/files, /etc/mkinitfs/files
## /usr/share/mkinitfs/files-extra, /etc/mkinitfs/files-extra
Files with the *.files* extension are read as a list of
files/directories. Each line is in the format:
```
<source path>:<destination path>
```
The source path is the location, at runtime, of the file or directory
which will be copied to the destination path within the initramfs
archive. Specifying a destination path, with *:<destination path>* is
optional. If it is omitted, then the source path will be used as the
destination path within the archive. The source and destination paths
are delimited by a *:* (colon.) Destination path is ignored if the source
path is a glob that returns more than 1 file. This may change in the future.
[[ *Line in .files*
:< Comment
| */usr/share/bazz*
: File or directory */usr/share/bazz* would be added to the archive under */usr/share/bazz*
| */usr/share/bazz:/bazz*
: File or directory */usr/share/bazz* would be added to the archive under */bazz*
| */root/something/\**
: Everything under */root/something* would be added to the archive under */root/something*
| */etc/foo/\*/bazz:/foo*
: Anything that matches the glob will be installed under the source path in the archive. For example, */etc/foo/bar/bazz* would be installed at */etc/foo/bar/bazz* in the archive. The destination path is ignored.
It's possible to overwrite file/directory destinations from
configuration in */usr/share/mkinitfs* by specifying the same source
path(s) under the relevant directory in */etc/mkinitfs*, and changing
the destination path.
## /usr/share/mkinitfs/hooks, /etc/mkinitfs/hooks
## /usr/share/mkinitfs/hooks-extra*, /etc/mkinitfs/hooks-extra
Any files listed under these directories are copied as-is into the
relevant archives. Hooks are generally script files, but how they are
treated in the initramfs is entirely up to whatever init script is run
there on boot.
Hooks are installed in the initramfs under the */hooks* directory, and
under */hooks-extra* for the initramfs-extra.
## /usr/share/mkinitfs/modules, /etc/mkinitfs/modules
## /usr/share/mkinitfs/modules-extra, /etc/mkinitfs/modules-extra
Files with the *.modules* extention in these directories are lists of
kernel modules to include in the initramfs. Individual modules and
directories can be listed in the files here. Globbing is also supported.
Modules are installed in the initramfs archive under the same path they
exist on the system where mkinitfs is executed.
## /usr/share/mkinitfs/dirs, /etc/mkinitfs/dirs
Files with the *.dirs* extension in these directories are lists of
directories to create within the initramfs. There is no *-extra* variant,
since directories are of negligible size.
# BOOT-DEPLOY
After generating archives, mkinitfs will execute *boot-deploy*, using *$PATH* to
search for the app. The following commandline options are passed to it:
*-i* <initramfs filename>
Currently this is hardcoded to be "initramfs"
*-k* <kernel filename>
*-d* <work directory>
Path to the directory containing the build artifacts from mkinitfs.
*-o* <destination directory>
Path to the directory that boot-deploy should use as its root when
installing files.
*initramfs-extra*
This string is the filename of the initramfs-extra archive.
# AUTHORS
*Clayton Craft* <clayton@craftyguy.net>

6
go.mod
View File

@@ -1,10 +1,10 @@
module gitlab.com/postmarketOS/postmarketos-mkinitfs
go 1.16
go 1.20
require (
github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e
github.com/klauspost/compress v1.13.3 // indirect
github.com/klauspost/pgzip v1.2.5
github.com/klauspost/compress v1.15.12
github.com/ulikunitz/xz v0.5.10
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c
)

9
go.sum
View File

@@ -1,9 +1,8 @@
github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e h1:hHg27A0RSSp2Om9lubZpiMgVbvn39bsUmW9U5h0twqc=
github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e/go.mod h1:oDpT4efm8tSYHXV5tHSdRvBet/b/QzxZ+XyyPehvm3A=
github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/klauspost/compress v1.13.3 h1:BtAvtV1+h0YwSVwWoYXMREPpYu9VzTJ9QDI1TEg/iQQ=
github.com/klauspost/compress v1.13.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE=
github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/klauspost/compress v1.15.12 h1:YClS/PImqYbn+UILDnqxQCZ3RehC9N318SU3kElDUEM=
github.com/klauspost/compress v1.15.12/go.mod h1:QPwzmACJjUTFsnSHH934V6woptycfrDDJnH7hvFVbGM=
github.com/ulikunitz/xz v0.5.10 h1:t92gobL9l3HE202wg3rlk19F6X+JOxl9BBrCCMYEYd8=
github.com/ulikunitz/xz v0.5.10/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c h1:F1jZWGFhYfh0Ci55sIpILtKKK8p3i2/krTr0H1rg74I=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

View File

@@ -5,7 +5,7 @@ package archive
import (
"bytes"
"compress/flate"
"compress/gzip"
"fmt"
"io"
"log"
@@ -17,29 +17,55 @@ import (
"syscall"
"github.com/cavaliercoder/go-cpio"
"github.com/klauspost/pgzip"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/pkgs/misc"
"github.com/klauspost/compress/zstd"
"github.com/ulikunitz/xz"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/osutil"
)
type CompressFormat string
const (
FormatGzip CompressFormat = "gzip"
FormatLzma CompressFormat = "lzma"
FormatZstd CompressFormat = "zstd"
FormatNone CompressFormat = "none"
)
type CompressLevel string
const (
// Mapped to the "default" level for the given format
LevelDefault CompressLevel = "default"
// Maps to the fastest compression level for the given format
LevelFast CompressLevel = "fast"
// Maps to the best compression level for the given format
LevelBest CompressLevel = "best"
)
type Archive struct {
items archiveItems
cpioWriter *cpio.Writer
buf *bytes.Buffer
cpioWriter *cpio.Writer
buf *bytes.Buffer
compress_format CompressFormat
compress_level CompressLevel
items archiveItems
}
func New() (*Archive, error) {
func New(format CompressFormat, level CompressLevel) *Archive {
buf := new(bytes.Buffer)
archive := &Archive{
cpioWriter: cpio.NewWriter(buf),
buf: buf,
cpioWriter: cpio.NewWriter(buf),
buf: buf,
compress_format: format,
compress_level: level,
}
return archive, nil
return archive
}
type archiveItem struct {
sourcePath string
header *cpio.Header
sourcePath string
}
type archiveItems struct {
@@ -47,9 +73,50 @@ type archiveItems struct {
sync.RWMutex
}
// ExtractFormatLevel parses the given string in the format format[:level],
// where :level is one of CompressLevel consts. If level is omitted from the
// string, or if it can't be parsed, the level is set to the default level for
// the given format. If format is unknown, gzip is selected. This function is
// designed to always return something usable within this package.
func ExtractFormatLevel(s string) (format CompressFormat, level CompressLevel) {
f, l, found := strings.Cut(s, ":")
if !found {
l = "default"
}
level = CompressLevel(strings.ToLower(l))
format = CompressFormat(strings.ToLower(f))
switch level {
}
switch level {
case LevelBest:
case LevelDefault:
case LevelFast:
default:
log.Print("Unknown or no compression level set, using default")
level = LevelDefault
}
switch format {
case FormatGzip:
case FormatLzma:
log.Println("Format lzma doesn't support a compression level, using default settings")
level = LevelDefault
case FormatNone:
case FormatZstd:
default:
log.Print("Unknown or no compression format set, using gzip")
format = FormatGzip
}
return
}
// Adds the given item to the archiveItems, only if it doesn't already exist in
// the list. The items are kept sorted in ascending order.
func (a *archiveItems) Add(item archiveItem) {
func (a *archiveItems) add(item archiveItem) {
a.Lock()
defer a.Unlock()
@@ -117,11 +184,16 @@ func (archive *Archive) Write(path string, mode os.FileMode) error {
return nil
}
// Adds the given items in the map to the archive. The map format is {source path:dest path}.
// Internally this just calls AddItem on each key,value pair in the map.
func (archive *Archive) AddItems(paths map[string]string) error {
for s, d := range paths {
if err := archive.AddItem(s, d); err != nil {
// AddItems adds the given items in the map to the archive. The map format is
// {source path:dest path}. Internally this just calls AddItem on each
// key,value pair in the map.
func (archive *Archive) AddItems(flister filelist.FileLister) error {
list, err := flister.List()
if err != nil {
return err
}
for i := range list.IterItems() {
if err := archive.AddItem(i.Source, i.Dest); err != nil {
return err
}
}
@@ -170,7 +242,7 @@ func (archive *Archive) addFile(source string, dest string) error {
destFilename := strings.TrimPrefix(dest, "/")
archive.items.Add(archiveItem{
archive.items.add(archiveItem{
sourcePath: source,
header: &cpio.Header{
Name: destFilename,
@@ -186,21 +258,18 @@ func (archive *Archive) addFile(source string, dest string) error {
}
// make sure target is an absolute path
if !filepath.IsAbs(target) {
target, err = misc.RelativeSymlinkTargetToDir(target, filepath.Dir(source))
target, err = osutil.RelativeSymlinkTargetToDir(target, filepath.Dir(source))
if err != nil {
return err
}
}
// TODO: add verbose mode, print stuff like this:
// log.Printf("symlink: %q, target: %q", file, target)
// write symlink target
err = archive.addFile(target, target)
return err
}
destFilename := strings.TrimPrefix(dest, "/")
archive.items.Add(archiveItem{
archive.items.add(archiveItem{
sourcePath: source,
header: &cpio.Header{
Name: destFilename,
@@ -213,23 +282,60 @@ func (archive *Archive) addFile(source string, dest string) error {
return nil
}
func (archive *Archive) writeCompressed(path string, mode os.FileMode) error {
// TODO: support other compression formats, based on deviceinfo
func (archive *Archive) writeCompressed(path string, mode os.FileMode) (err error) {
var compressor io.WriteCloser
defer func() {
e := compressor.Close()
if e != nil && err == nil {
err = e
}
}()
fd, err := os.Create(path)
if err != nil {
return err
}
// Note: fd.Close omitted since it'll be closed in "compressor"
gz, err := pgzip.NewWriterLevel(fd, flate.BestSpeed)
if err != nil {
return err
switch archive.compress_format {
case FormatGzip:
level := gzip.DefaultCompression
switch archive.compress_level {
case LevelBest:
level = gzip.BestCompression
case LevelFast:
level = gzip.BestSpeed
}
compressor, err = gzip.NewWriterLevel(fd, level)
if err != nil {
return err
}
case FormatLzma:
compressor, err = xz.NewWriter(fd)
if err != nil {
return err
}
case FormatNone:
compressor = fd
case FormatZstd:
level := zstd.SpeedDefault
switch archive.compress_level {
case LevelBest:
level = zstd.SpeedBestCompression
case LevelFast:
level = zstd.SpeedFastest
}
compressor, err = zstd.NewWriter(fd, zstd.WithEncoderLevel(level))
if err != nil {
return err
}
default:
log.Print("Unknown or no compression format set, using gzip")
compressor = gzip.NewWriter(fd)
}
if _, err = io.Copy(gz, archive.buf); err != nil {
return err
}
if err := gz.Close(); err != nil {
if _, err = io.Copy(compressor, archive.buf); err != nil {
return err
}
@@ -295,7 +401,7 @@ func (archive *Archive) addDir(dir string) error {
subdirs := strings.Split(strings.TrimPrefix(dir, "/"), "/")
for i, subdir := range subdirs {
path := filepath.Join(strings.Join(subdirs[:i], "/"), subdir)
archive.items.Add(archiveItem{
archive.items.add(archiveItem{
sourcePath: path,
header: &cpio.Header{
Name: path,

View File

@@ -180,10 +180,93 @@ func TestArchiveItemsAdd(t *testing.T) {
for _, st := range subtests {
t.Run(st.name, func(t *testing.T) {
a := archiveItems{items: st.inItems}
a.Add(st.inItem)
a.add(st.inItem)
if !reflect.DeepEqual(st.expected, a.items) {
t.Fatal("expected:", st.expected, " got: ", a.items)
}
})
}
}
func TestExtractFormatLevel(t *testing.T) {
tests := []struct {
name string
in string
expectedFormat CompressFormat
expectedLevel CompressLevel
}{
{
name: "gzip, default level",
in: "gzip:default",
expectedFormat: FormatGzip,
expectedLevel: LevelDefault,
},
{
name: "unknown format, level 12",
in: "pear:12",
expectedFormat: FormatGzip,
expectedLevel: LevelDefault,
},
{
name: "zstd, level not given",
in: "zstd",
expectedFormat: FormatZstd,
expectedLevel: LevelDefault,
},
{
name: "zstd, invalid level 'fast:'",
in: "zstd:fast:",
expectedFormat: FormatZstd,
expectedLevel: LevelDefault,
},
{
name: "zstd, best",
in: "zstd:best",
expectedFormat: FormatZstd,
expectedLevel: LevelBest,
},
{
name: "zstd, level empty :",
in: "zstd:",
expectedFormat: FormatZstd,
expectedLevel: LevelDefault,
},
{
name: "gzip, best",
in: "gzip:best",
expectedFormat: FormatGzip,
expectedLevel: LevelBest,
},
{
name: "<empty>, <empty>",
in: "",
expectedFormat: FormatGzip,
expectedLevel: LevelDefault,
},
{
name: "lzma, fast",
in: "lzma:fast",
expectedFormat: FormatLzma,
expectedLevel: LevelDefault,
},
{
name: "none",
in: "none",
expectedFormat: FormatNone,
expectedLevel: LevelDefault,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
format, level := ExtractFormatLevel(test.in)
if format != test.expectedFormat {
t.Fatal("format expected: ", test.expectedFormat, " got: ", format)
}
if level != test.expectedLevel {
t.Fatal("level expected: ", test.expectedLevel, " got: ", level)
}
})
}
}

View File

@@ -0,0 +1,153 @@
package bootdeploy
import (
"errors"
"fmt"
"io"
"log"
"os"
"os/exec"
"path"
"path/filepath"
"strings"
)
type BootDeploy struct {
inDir string
outDir string
ubootBoardname string
}
// New returns a new BootDeploy, which then runs:
//
// boot-deploy -d indir -o outDir
//
// ubootBoardname is used for copying in some u-boot files prior to running
// boot-deploy. This is optional, passing an empty string is ok if this is not
// needed.
func New(inDir, outDir, ubootBoardname string) *BootDeploy {
return &BootDeploy{
inDir: inDir,
outDir: outDir,
ubootBoardname: ubootBoardname,
}
}
func (b *BootDeploy) Run() error {
if err := copyUbootFiles(b.inDir, b.ubootBoardname); errors.Is(err, os.ErrNotExist) {
log.Println("u-boot files copying skipped: ", err)
} else {
if err != nil {
log.Fatal("copyUbootFiles: ", err)
}
}
return bootDeploy(b.inDir, b.outDir)
}
func bootDeploy(workDir string, outDir string) error {
// boot-deploy expects the kernel to be in the same dir as initramfs.
// Assume that the kernel is in the output dir...
kernels, _ := filepath.Glob(filepath.Join(outDir, "vmlinuz*"))
if len(kernels) == 0 {
return errors.New("Unable to find any kernels at " + filepath.Join(outDir, "vmlinuz*"))
}
// Pick a kernel that does not have suffixes added by boot-deploy
var kernFile string
for _, f := range kernels {
if strings.HasSuffix(f, "-dtb") || strings.HasSuffix(f, "-mtk") {
continue
}
kernFile = f
break
}
kernFd, err := os.Open(kernFile)
if err != nil {
return err
}
defer kernFd.Close()
kernFilename := path.Base(kernFile)
kernFileCopy, err := os.Create(filepath.Join(workDir, kernFilename))
if err != nil {
return err
}
if _, err = io.Copy(kernFileCopy, kernFd); err != nil {
return err
}
if err := kernFileCopy.Close(); err != nil {
return fmt.Errorf("error closing %s: %w", kernFilename, err)
}
// boot-deploy -i initramfs -k vmlinuz-postmarketos-rockchip -d /tmp/cpio -o /tmp/foo initramfs-extra
cmd := exec.Command("boot-deploy",
"-i", "initramfs",
"-k", kernFilename,
"-d", workDir,
"-o", outDir,
"initramfs-extra")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
return err
}
return nil
}
// Copy copies the file at srcFile path to a new file at dstFile path
func copy(srcFile, dstFile string) error {
out, err := os.Create(dstFile)
if err != nil {
return err
}
defer func() {
errClose := out.Close()
if err == nil {
err = errClose
}
}()
in, err := os.Open(srcFile)
if err != nil {
return err
}
defer in.Close()
_, err = io.Copy(out, in)
if err != nil {
return err
}
return nil
}
// copyUbootFiles uses deviceinfo_uboot_boardname to copy u-boot files required
// for running boot-deploy
func copyUbootFiles(path, ubootBoardname string) error {
if ubootBoardname == "" {
return nil
}
srcDir := filepath.Join("/usr/share/u-boot", ubootBoardname)
entries, err := os.ReadDir(srcDir)
if err != nil {
return err
}
for _, entry := range entries {
sourcePath := filepath.Join(srcDir, entry.Name())
destPath := filepath.Join(path, entry.Name())
if err := copy(sourcePath, destPath); err != nil {
return err
}
}
return nil
}

View File

@@ -0,0 +1,91 @@
package filelist
import (
"fmt"
"path/filepath"
"sync"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/misc"
)
type FileLister interface {
List() (*FileList, error)
}
type File struct {
Source string
Dest string
}
type FileList struct {
m map[string]string
sync.RWMutex
}
func NewFileList() *FileList {
return &FileList{
m: make(map[string]string),
}
}
func (f *FileList) Add(src string, dest string) {
f.Lock()
defer f.Unlock()
f.m[src] = dest
}
func (f *FileList) Get(src string) (string, bool) {
f.RLock()
defer f.RUnlock()
dest, found := f.m[src]
return dest, found
}
// Import copies in the contents of src. If a source path already exists when
// importing, then the destination path is updated with the new value.
func (f *FileList) Import(src *FileList) {
for i := range src.IterItems() {
f.Add(i.Source, i.Dest)
}
}
func (f *FileList) AddGlobbed(src string, dest string) error {
fFiles, err := misc.GetFiles([]string{src}, true)
if err != nil {
return fmt.Errorf("unable to add %q: %w", src, err)
}
// loop over all returned files from GetFile
for _, file := range fFiles {
if len(fFiles) > 1 {
// Glob with arbitrary subdirectories, so we need to
// remove the src path and prepend the dest path
f.Add(file, filepath.Join(dest, file[len(src):]))
} else {
// dest path specified, and only 1 file
f.Add(file, dest)
}
}
return nil
}
// iterate through the list and and send each one as a new File over the
// returned channel
func (f *FileList) IterItems() <-chan File {
ch := make(chan File)
go func() {
f.RLock()
defer f.RUnlock()
for src, dest := range f.m {
ch <- File{
Source: src,
Dest: dest,
}
}
close(ch)
}()
return ch
}

View File

@@ -0,0 +1,51 @@
package hookdirs
import (
"bufio"
"fmt"
"log"
"os"
"path/filepath"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
)
type HookDirs struct {
path string
}
// New returns a new HookDirs that will use the given path to provide a list
// of directories use.
func New(path string) *HookDirs {
return &HookDirs{
path: path,
}
}
func (h *HookDirs) List() (*filelist.FileList, error) {
log.Printf("- Searching for directories specified in %s", h.path)
files := filelist.NewFileList()
fileInfo, err := os.ReadDir(h.path)
if err != nil {
log.Println("-- Unable to find dir, skipping...")
return files, nil
}
for _, file := range fileInfo {
path := filepath.Join(h.path, file.Name())
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("getHookDirs: unable to open hook file: %w", err)
}
defer f.Close()
log.Printf("-- Creating directories from: %s\n", path)
s := bufio.NewScanner(f)
for s.Scan() {
dir := s.Text()
files.Add(dir, dir)
}
}
return files, nil
}

View File

@@ -0,0 +1,73 @@
package hookfiles
import (
"bufio"
"fmt"
"io"
"log"
"os"
"path/filepath"
"strings"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
)
type HookFiles struct {
filePath string
}
// New returns a new HookFiles that will use the given path to provide a list
// of files + any binary dependencies they might have.
func New(filePath string) *HookFiles {
return &HookFiles{
filePath: filePath,
}
}
func (h *HookFiles) List() (*filelist.FileList, error) {
log.Printf("- Searching for file lists from %s", h.filePath)
files := filelist.NewFileList()
fileInfo, err := os.ReadDir(h.filePath)
if err != nil {
log.Println("-- Unable to find dir, skipping...")
return files, nil
}
for _, file := range fileInfo {
path := filepath.Join(h.filePath, file.Name())
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("getHookFiles: unable to open hook file: %w", err)
}
defer f.Close()
log.Printf("-- Including files from: %s\n", path)
if list, err := slurpFiles(f); err != nil {
return nil, fmt.Errorf("hookfiles: unable to process hook file %q: %w", path, err)
} else {
files.Import(list)
}
}
return files, nil
}
func slurpFiles(fd io.Reader) (*filelist.FileList, error) {
files := filelist.NewFileList()
s := bufio.NewScanner(fd)
for s.Scan() {
src, dest, has_dest := strings.Cut(s.Text(), ":")
if !has_dest {
dest = src
}
err := files.AddGlobbed(src, dest)
if err != nil {
return nil, err
}
}
return files, s.Err()
}

View File

@@ -0,0 +1,47 @@
package hookscripts
import (
"log"
"os"
"path/filepath"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
)
type HookScripts struct {
destPath string
scriptsDir string
}
// New returns a new HookScripts that will use the given path to provide a list
// of script files. The destination for each script it set to destPath, using
// the original file name.
func New(scriptsDir string, destPath string) *HookScripts {
return &HookScripts{
destPath: destPath,
scriptsDir: scriptsDir,
}
}
func (h *HookScripts) List() (*filelist.FileList, error) {
log.Printf("- Searching for hook scripts from %s", h.scriptsDir)
files := filelist.NewFileList()
fileInfo, err := os.ReadDir(h.scriptsDir)
if err != nil {
log.Println("-- Unable to find dir, skipping...")
return files, nil
}
for _, file := range fileInfo {
path := filepath.Join(h.scriptsDir, file.Name())
if file.IsDir() {
log.Printf("-- Including dir %s\n", path)
files.AddGlobbed(filepath.Join(path, "*"), filepath.Join(h.destPath, file.Name()))
} else {
log.Printf("-- Including script: %s\n", path)
files.Add(path, filepath.Join(h.destPath, file.Name()))
}
}
return files, nil
}

View File

@@ -0,0 +1,34 @@
package initramfs
import (
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
)
// Initramfs allows building arbitrarily complex lists of features, by slurping
// up types that implement FileLister (which includes this type! yippee) and
// combining the output from them.
type Initramfs struct {
features []filelist.FileLister
}
// New returns a new Initramfs that generate a list of files based on the given
// list of FileListers.
func New(features []filelist.FileLister) *Initramfs {
return &Initramfs{
features: features,
}
}
func (i *Initramfs) List() (*filelist.FileList, error) {
files := filelist.NewFileList()
for _, f := range i.features {
list, err := f.List()
if err != nil {
return nil, err
}
files.Import(list)
}
return files, nil
}

View File

@@ -0,0 +1,219 @@
package modules
import (
"bufio"
"fmt"
"io"
"log"
"os"
"path/filepath"
"regexp"
"strings"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/misc"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/osutil"
)
type Modules struct {
modulesListPath string
modulesList []string
}
// New returns a new Modules that will use the given moduleto provide a list
// of script files.
func New(modulesList []string, modulesListPath string) *Modules {
return &Modules{
modulesList: modulesList,
modulesListPath: modulesListPath,
}
}
func (m *Modules) List() (*filelist.FileList, error) {
kernVer, err := osutil.GetKernelVersion()
if err != nil {
return nil, err
}
files := filelist.NewFileList()
modDir := filepath.Join("/lib/modules", kernVer)
if exists, err := misc.Exists(modDir); !exists {
// dir /lib/modules/<kernel> if kernel built without module support, so just print a message
log.Printf("-- kernel module directory not found: %q, not including modules", modDir)
return files, nil
} else if err != nil {
return nil, fmt.Errorf("received unexpected error when getting status for %q: %w", modDir, err)
}
// modules.* required by modprobe
modprobeFiles, _ := filepath.Glob(filepath.Join(modDir, "modules.*"))
for _, file := range modprobeFiles {
files.Add(file, file)
}
// slurp up given list of modules
if len(m.modulesList) > 0 {
log.Printf("-- Including kernel modules from deviceinfo")
for _, module := range m.modulesList {
if modFilelist, err := getModule(module, modDir); err != nil {
return nil, fmt.Errorf("unable to get modules from deviceinfo: %w", err)
} else {
for _, file := range modFilelist {
files.Add(file, file)
}
}
}
}
// slurp up modules from lists in modulesListPath
log.Printf("- Searching for kernel modules from %s", m.modulesListPath)
fileInfo, err := os.ReadDir(m.modulesListPath)
if err != nil {
return files, nil
}
for _, file := range fileInfo {
path := filepath.Join(m.modulesListPath, file.Name())
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("unable to open module list file %q: %w", path, err)
}
defer f.Close()
log.Printf("-- Including modules from: %s\n", path)
if list, err := slurpModules(f, modDir); err != nil {
return nil, fmt.Errorf("unable to process module list file %q: %w", path, err)
} else {
files.Import(list)
}
}
return files, nil
}
func slurpModules(fd io.Reader, modDir string) (*filelist.FileList, error) {
files := filelist.NewFileList()
s := bufio.NewScanner(fd)
for s.Scan() {
line := s.Text()
dir, file := filepath.Split(line)
if file == "" {
// item is a directory
dir = filepath.Join(modDir, dir)
dirs, _ := filepath.Glob(dir)
for _, d := range dirs {
if modFilelist, err := getModulesInDir(d); err != nil {
return nil, fmt.Errorf("unable to get modules dir %q: %w", d, err)
} else {
for _, file := range modFilelist {
files.Add(file, file)
}
}
}
} else if dir == "" {
// item is a module name
if modFilelist, err := getModule(s.Text(), modDir); err != nil {
return nil, fmt.Errorf("unable to get module file %q: %w", s.Text(), err)
} else {
for _, file := range modFilelist {
files.Add(file, file)
}
}
} else {
log.Printf("Unknown module entry: %q", line)
}
}
return files, s.Err()
}
func getModulesInDir(modPath string) (files []string, err error) {
err = filepath.Walk(modPath, func(path string, _ os.FileInfo, err error) error {
if err != nil {
// Unable to walk path
return err
}
if filepath.Ext(path) != ".ko" && filepath.Ext(path) != ".xz" {
return nil
}
files = append(files, path)
return nil
})
if err != nil {
return nil, err
}
return
}
// Given a module name, e.g. 'dwc_wdt', resolve the full path to the module
// file and all of its dependencies.
// Note: it's not necessarily fatal if the module is not found, since it may
// have been built into the kernel
func getModule(modName string, modDir string) (files []string, err error) {
modDep := filepath.Join(modDir, "modules.dep")
if exists, err := misc.Exists(modDep); !exists {
return nil, fmt.Errorf("kernel module.dep not found: %s", modDir)
} else if err != nil {
return nil, fmt.Errorf("received unexpected error when getting module.dep status: %w", err)
}
fd, err := os.Open(modDep)
if err != nil {
return nil, fmt.Errorf("unable to open modules.dep: %w", err)
}
defer fd.Close()
deps, err := getModuleDeps(modName, fd)
if err != nil {
return nil, err
}
for _, dep := range deps {
p := filepath.Join(modDir, dep)
if exists, err := misc.Exists(p); !exists {
return nil, fmt.Errorf("tried to include a module that doesn't exist in the modules directory (%s): %s", modDir, p)
} else if err != nil {
return nil, fmt.Errorf("received unexpected error when getting status for %q: %w", p, err)
}
files = append(files, p)
}
return
}
// Get the canonicalized name for the module as represented in the given modules.dep io.reader
func getModuleDeps(modName string, modulesDep io.Reader) ([]string, error) {
var deps []string
// split the module name on - and/or _, build a regex for matching
splitRe := regexp.MustCompile("[-_]+")
modNameReStr := splitRe.ReplaceAllString(modName, "[-_]+")
re := regexp.MustCompile("^" + modNameReStr + "$")
s := bufio.NewScanner(modulesDep)
for s.Scan() {
fields := strings.Fields(s.Text())
if len(fields) == 0 {
continue
}
fields[0] = strings.TrimSuffix(fields[0], ":")
found := re.FindAll([]byte(filepath.Base(stripExts(fields[0]))), -1)
if len(found) > 0 {
deps = append(deps, fields...)
break
}
}
if err := s.Err(); err != nil {
log.Print("Unable to get module + dependencies: ", modName)
return deps, err
}
return deps, nil
}
func stripExts(file string) string {
return strings.Split(file, ".")[0]
}

View File

@@ -1,7 +1,7 @@
// Copyright 2021 Clayton Craft <clayton@craftyguy.net>
// Copyright 2023 Clayton Craft <clayton@craftyguy.net>
// SPDX-License-Identifier: GPL-3.0-or-later
package main
package modules
import (
"strings"
@@ -27,18 +27,6 @@ func TestStripExts(t *testing.T) {
}
}
func stringSlicesEqual(a []string, b []string) bool {
if len(a) != len(b) {
return false
}
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}
var testModuleDep string = `
kernel/sound/soc/codecs/snd-soc-msm8916-digital.ko:
kernel/net/sched/act_ipt.ko.xz: kernel/net/netfilter/x_tables.ko.xz
@@ -80,3 +68,15 @@ func TestGetModuleDeps(t *testing.T) {
}
}
}
func stringSlicesEqual(a []string, b []string) bool {
if len(a) != len(b) {
return false
}
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}

View File

@@ -0,0 +1,158 @@
package osksdl
import (
"bufio"
"fmt"
"log"
"os"
"path/filepath"
"strings"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/filelist"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/misc"
)
type OskSdl struct {
mesaDriver string
}
// New returns a new HookScripts that will use the given path to provide a list
// of script files.
func New(mesaDriverName string) *OskSdl {
return &OskSdl{
mesaDriver: mesaDriverName,
}
}
// Get a list of files and their dependencies related to supporting rootfs full
// disk (d)encryption
func (s *OskSdl) List() (*filelist.FileList, error) {
files := filelist.NewFileList()
if exists, err := misc.Exists("/usr/bin/osk-sdl"); !exists {
return files, nil
} else if err != nil {
return files, fmt.Errorf("received unexpected error when getting status for %q: %w", "/usr/bin/osk-sdl", err)
}
log.Println("- Including osk-sdl support")
confFiles := []string{
"/etc/osk.conf",
"/etc/ts.conf",
"/etc/pointercal",
"/etc/fb.modes",
"/etc/directfbrc",
}
confFileList, err := misc.GetFiles(confFiles, false)
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
}
for _, file := range confFileList {
files.Add(file, file)
}
// osk-sdl
oskFiles := []string{
"/usr/bin/osk-sdl",
"/sbin/cryptsetup",
"/usr/lib/libGL.so.1",
}
if oskFileList, err := misc.GetFiles(oskFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
for _, file := range oskFileList {
files.Add(file, file)
}
}
fontFile, err := getOskConfFontPath("/etc/osk.conf")
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add file %q: %w", fontFile, err)
}
files.Add(fontFile, fontFile)
// Directfb
dfbFiles := []string{}
err = filepath.Walk("/usr/lib/directfb-1.7-7", func(path string, f os.FileInfo, err error) error {
if filepath.Ext(path) == ".so" {
dfbFiles = append(dfbFiles, path)
}
return nil
})
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add file %w", err)
}
if dfbFileList, err := misc.GetFiles(dfbFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
for _, file := range dfbFileList {
files.Add(file, file)
}
}
// tslib
tslibFiles := []string{}
err = filepath.Walk("/usr/lib/ts", func(path string, f os.FileInfo, err error) error {
if filepath.Ext(path) == ".so" {
tslibFiles = append(tslibFiles, path)
}
return nil
})
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add file: %w", err)
}
libts, _ := filepath.Glob("/usr/lib/libts*")
tslibFiles = append(tslibFiles, libts...)
if tslibFileList, err := misc.GetFiles(tslibFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
for _, file := range tslibFileList {
files.Add(file, file)
}
}
// mesa hw accel
if s.mesaDriver != "" {
mesaFiles := []string{
"/usr/lib/libEGL.so.1",
"/usr/lib/libGLESv2.so.2",
"/usr/lib/libgbm.so.1",
"/usr/lib/libudev.so.1",
"/usr/lib/xorg/modules/dri/" + s.mesaDriver + "_dri.so",
}
if mesaFileList, err := misc.GetFiles(mesaFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
for _, file := range mesaFileList {
files.Add(file, file)
}
}
}
return files, nil
}
func getOskConfFontPath(oskConfPath string) (string, error) {
var path string
f, err := os.Open(oskConfPath)
if err != nil {
return path, err
}
defer f.Close()
s := bufio.NewScanner(f)
for s.Scan() {
fields := strings.Fields(s.Text())
// "key = val" is 3 fields
if len(fields) > 2 && fields[0] == "keyboard-font" {
path = fields[2]
}
}
if exists, err := misc.Exists(path); !exists {
return path, fmt.Errorf("unable to find font: %s", path)
} else if err != nil {
return path, fmt.Errorf("received unexpected error when getting status for %q: %w", path, err)
}
return path, nil
}

165
internal/misc/getfiles.go Normal file
View File

@@ -0,0 +1,165 @@
package misc
import (
"debug/elf"
"fmt"
"os"
"path/filepath"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/internal/osutil"
)
func GetFiles(list []string, required bool) (files []string, err error) {
for _, file := range list {
filelist, err := getFile(file, required)
if err != nil {
return nil, err
}
files = append(files, filelist...)
}
files = RemoveDuplicates(files)
return
}
func getFile(file string, required bool) (files []string, err error) {
// Expand glob expression
expanded, err := filepath.Glob(file)
if err != nil {
return
}
if len(expanded) > 0 && expanded[0] != file {
for _, path := range expanded {
if globFiles, err := getFile(path, required); err != nil {
return files, err
} else {
files = append(files, globFiles...)
}
}
return RemoveDuplicates(files), nil
}
fileInfo, err := os.Stat(file)
if err != nil {
if required {
return files, fmt.Errorf("getFile: failed to stat file %q: %w", file, err)
}
return files, nil
}
if fileInfo.IsDir() {
// Recurse over directory contents
err := filepath.Walk(file, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
if f.IsDir() {
return nil
}
newFiles, err := getFile(path, required)
if err != nil {
return err
}
files = append(files, newFiles...)
return nil
})
if err != nil {
return files, err
}
} else {
files = append(files, file)
// get dependencies for binaries
if _, err := elf.Open(file); err == nil {
if binaryDepFiles, err := getBinaryDeps(file); err != nil {
return files, err
} else {
files = append(files, binaryDepFiles...)
}
}
}
files = RemoveDuplicates(files)
return
}
func getDeps(file string, parents map[string]struct{}) (files []string, err error) {
if _, found := parents[file]; found {
return
}
// get dependencies for binaries
fd, err := elf.Open(file)
if err != nil {
return nil, fmt.Errorf("getDeps: unable to open elf binary %q: %w", file, err)
}
libs, _ := fd.ImportedLibraries()
fd.Close()
files = append(files, file)
parents[file] = struct{}{}
if len(libs) == 0 {
return
}
// we don't recursively search these paths for performance reasons
libdirGlobs := []string{
"/usr/lib",
"/lib",
"/usr/lib/expect*",
}
for _, lib := range libs {
found := false
findDepLoop:
for _, libdirGlob := range libdirGlobs {
libdirs, _ := filepath.Glob(libdirGlob)
for _, libdir := range libdirs {
path := filepath.Join(libdir, lib)
if _, err := os.Stat(path); err == nil {
binaryDepFiles, err := getDeps(path, parents)
if err != nil {
return nil, err
}
files = append(files, binaryDepFiles...)
files = append(files, path)
found = true
break findDepLoop
}
}
}
if !found {
return nil, fmt.Errorf("getDeps: unable to locate dependency for %q: %s", file, lib)
}
}
return
}
// Recursively list all dependencies for a given ELF binary
func getBinaryDeps(file string) ([]string, error) {
// if file is a symlink, resolve dependencies for target
fileStat, err := os.Lstat(file)
if err != nil {
return nil, fmt.Errorf("getBinaryDeps: failed to stat file %q: %w", file, err)
}
// Symlink: write symlink to archive then set 'file' to link target
if fileStat.Mode()&os.ModeSymlink != 0 {
target, err := os.Readlink(file)
if err != nil {
return nil, fmt.Errorf("getBinaryDeps: unable to read symlink %q: %w", file, err)
}
if !filepath.IsAbs(target) {
target, err = osutil.RelativeSymlinkTargetToDir(target, filepath.Dir(file))
if err != nil {
return nil, err
}
}
file = target
}
return getDeps(file, make(map[string]struct{}))
}

65
internal/misc/misc.go Normal file
View File

@@ -0,0 +1,65 @@
// Copyright 2022 Clayton Craft <clayton@craftyguy.net>
// SPDX-License-Identifier: GPL-3.0-or-later
package misc
import (
"errors"
"log"
"os"
"time"
)
// Merge the contents of "b" into "a", overwriting any previously existing keys
// in "a"
func Merge(a map[string]string, b map[string]string) {
for k, v := range b {
a[k] = v
}
}
// Removes duplicate entries from the given string slice and returns a slice
// with the unique values
func RemoveDuplicates(in []string) (out []string) {
// use a map to "remove" duplicates. the value in the map is totally
// irrelevant
outMap := make(map[string]bool)
for _, s := range in {
if ok := outMap[s]; !ok {
outMap[s] = true
}
}
out = make([]string, 0, len(outMap))
for k := range outMap {
out = append(out, k)
}
return
}
// Prints the execution time of a function, not meant to be very
// sensitive/accurate, but good enough to gauge rough run times.
// Meant to be called as:
//
// defer misc.TimeFunc(time.Now(), "foo")
func TimeFunc(start time.Time, name string) {
elapsed := time.Since(start)
log.Printf("%s completed in: %.2fs", name, elapsed.Seconds())
}
// Exists tests if the given file/dir exists or not. Returns any errors related
// to os.Stat if the type is *not* ErrNotExist. If an error is returned, then
// the value of the returned boolean cannot be trusted.
func Exists(file string) (bool, error) {
_, err := os.Stat(file)
if err == nil {
return true, nil
} else if errors.Is(err, os.ErrNotExist) {
// Don't return the error, the file doesn't exist which is OK
return false, nil
}
// Other errors from os.Stat returned here
return false, err
}

View File

@@ -1,13 +1,13 @@
// Copyright 2022 Clayton Craft <clayton@craftyguy.net>
// SPDX-License-Identifier: GPL-3.0-or-later
package misc
package osutil
import (
"golang.org/x/sys/unix"
"fmt"
"log"
"os"
"path/filepath"
"strings"
"golang.org/x/sys/unix"
)
// Converts a relative symlink target path (e.g. ../../lib/foo.so), that is
@@ -47,30 +47,28 @@ func FreeSpace(path string) (uint64, error) {
return size, nil
}
// Merge the contents of "b" into "a", overwriting any previously existing keys
// in "a"
func Merge(a map[string]string, b map[string]string) {
for k, v := range b {
a[k] = v
func getKernelReleaseFile() (string, error) {
files, _ := filepath.Glob("/usr/share/kernel/*/kernel.release")
// only one kernel flavor supported
if len(files) != 1 {
return "", fmt.Errorf("only one kernel release/flavor is supported, found: %q", files)
}
return files[0], nil
}
// Removes duplicate entries from the given string slice and returns a slice
// with the unique values
func RemoveDuplicates(in []string) (out []string) {
// use a map to "remove" duplicates. the value in the map is totally
// irrelevant
outMap := make(map[string]bool)
for _, s := range in {
if ok := outMap[s]; !ok {
outMap[s] = true
}
func GetKernelVersion() (string, error) {
var version string
releaseFile, err := getKernelReleaseFile()
if err != nil {
return version, err
}
out = make([]string, 0, len(outMap))
for k := range outMap {
out = append(out, k)
contents, err := os.ReadFile(releaseFile)
if err != nil {
return version, err
}
return
return strings.TrimSpace(string(contents)), nil
}

855
main.go
View File

@@ -1,855 +0,0 @@
// Copyright 2022 Clayton Craft <clayton@craftyguy.net>
// SPDX-License-Identifier: GPL-3.0-or-later
package main
import (
"bufio"
"debug/elf"
"errors"
"flag"
"fmt"
"io"
"log"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/pkgs/archive"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/pkgs/deviceinfo"
"gitlab.com/postmarketOS/postmarketos-mkinitfs/pkgs/misc"
)
func timeFunc(start time.Time, name string) {
elapsed := time.Since(start)
log.Printf("%s completed in: %s", name, elapsed)
}
func main() {
deviceinfoFile := "/etc/deviceinfo"
if !exists(deviceinfoFile) {
log.Print("NOTE: deviceinfo (from device package) not installed yet, " +
"not building the initramfs now (it should get built later " +
"automatically.)")
return
}
devinfo, err := deviceinfo.ReadDeviceinfo(deviceinfoFile)
if err != nil {
log.Fatal(err)
}
outDir := flag.String("d", "/boot", "Directory to output initfs(-extra) and other boot files")
flag.Parse()
defer timeFunc(time.Now(), "mkinitfs")
kernVer, err := getKernelVersion()
if err != nil {
log.Fatal(err)
}
// temporary working dir
workDir, err := os.MkdirTemp("", "mkinitfs")
if err != nil {
log.Fatal("Unable to create temporary work directory:", err)
}
defer os.RemoveAll(workDir)
log.Print("Generating for kernel version: ", kernVer)
log.Print("Output directory: ", *outDir)
if err := generateInitfs("initramfs", workDir, kernVer, devinfo); err != nil {
log.Fatal("generateInitfs: ", err)
}
if err := generateInitfsExtra("initramfs-extra", workDir, devinfo); err != nil {
log.Fatal("generateInitfsExtra: ", err)
}
if err := copyUbootFiles(workDir, devinfo); errors.Is(err, os.ErrNotExist) {
log.Println("u-boot files copying skipped: ", err)
} else {
if err != nil {
log.Fatal("copyUbootFiles: ", err)
}
}
// Final processing of initramfs / kernel is done by boot-deploy
if err := bootDeploy(workDir, *outDir); err != nil {
log.Fatal("bootDeploy: ", err)
}
}
func bootDeploy(workDir string, outDir string) error {
// boot-deploy expects the kernel to be in the same dir as initramfs.
// Assume that the kernel is in the output dir...
log.Print("== Using boot-deploy to finalize/install files ==")
kernels, _ := filepath.Glob(filepath.Join(outDir, "vmlinuz*"))
if len(kernels) == 0 {
return errors.New("Unable to find any kernels at " + filepath.Join(outDir, "vmlinuz*"))
}
// Pick a kernel that does not have suffixes added by boot-deploy
var kernFile string
for _, f := range kernels {
if strings.HasSuffix(f, "-dtb") || strings.HasSuffix(f, "-mtk") {
continue
}
kernFile = f
break
}
kernFd, err := os.Open(kernFile)
if err != nil {
return err
}
defer kernFd.Close()
kernFileCopy, err := os.Create(filepath.Join(workDir, "vmlinuz"))
if err != nil {
return err
}
if _, err = io.Copy(kernFileCopy, kernFd); err != nil {
return err
}
kernFileCopy.Close()
// boot-deploy -i initramfs -k vmlinuz-postmarketos-rockchip -d /tmp/cpio -o /tmp/foo initramfs-extra
cmd := exec.Command("boot-deploy",
"-i", "initramfs",
"-k", "vmlinuz",
"-d", workDir,
"-o", outDir,
"initramfs-extra")
if !exists(cmd.Path) {
return errors.New("boot-deploy command not found")
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
log.Print("'boot-deploy' command failed")
return err
}
return nil
}
func exists(file string) bool {
if _, err := os.Stat(file); err == nil {
return true
}
return false
}
func getHookFiles(filesdir string) (files []string, err error) {
fileInfo, err := os.ReadDir(filesdir)
if err != nil {
return nil, fmt.Errorf("getHookFiles: unable to read hook file dir: %w", err)
}
for _, file := range fileInfo {
path := filepath.Join(filesdir, file.Name())
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("getHookFiles: unable to open hook file: %w", err)
}
defer f.Close()
log.Printf("-- Including files from: %s\n", path)
s := bufio.NewScanner(f)
for s.Scan() {
if filelist, err := getFiles([]string{s.Text()}, true); err != nil {
return nil, fmt.Errorf("getHookFiles: unable to find file %q required by %q", s.Text(), path)
} else {
files = append(files, filelist...)
}
}
if err := s.Err(); err != nil {
return nil, fmt.Errorf("getHookFiles: uname to process hook file %q: %w", path, err)
}
}
return files, nil
}
// Recursively list all dependencies for a given ELF binary
func getBinaryDeps(file string) (files []string, err error) {
// if file is a symlink, resolve dependencies for target
fileStat, err := os.Lstat(file)
if err != nil {
return nil, fmt.Errorf("getBinaryDeps: failed to stat file %q: %w", file, err)
}
// Symlink: write symlink to archive then set 'file' to link target
if fileStat.Mode()&os.ModeSymlink != 0 {
target, err := os.Readlink(file)
if err != nil {
return nil, fmt.Errorf("getBinaryDeps: unable to read symlink %q: %w", file, err)
}
if !filepath.IsAbs(target) {
target, err = misc.RelativeSymlinkTargetToDir(target, filepath.Dir(file))
if err != nil {
return files, err
}
}
binaryDepFiles, err := getBinaryDeps(target)
if err != nil {
return files, err
}
files = append(files, binaryDepFiles...)
return files, err
}
// get dependencies for binaries
fd, err := elf.Open(file)
if err != nil {
return nil, fmt.Errorf("getBinaryDeps: unable to open elf binary %q: %w", file, err)
}
libs, _ := fd.ImportedLibraries()
fd.Close()
files = append(files, file)
if len(libs) == 0 {
return files, err
}
libdirs := []string{"/usr/lib", "/lib"}
for _, lib := range libs {
found := false
for _, libdir := range libdirs {
path := filepath.Join(libdir, lib)
if _, err := os.Stat(path); err == nil {
binaryDepFiles, err := getBinaryDeps(path)
if err != nil {
return files, err
}
files = append(files, binaryDepFiles...)
files = append(files, path)
found = true
break
}
}
if !found {
return nil, fmt.Errorf("getBinaryDeps: unable to locate dependency for %q: %s", file, lib)
}
}
return
}
func getFiles(list []string, required bool) (files []string, err error) {
for _, file := range list {
filelist, err := getFile(file, required)
if err != nil {
return nil, err
}
files = append(files, filelist...)
}
files = misc.RemoveDuplicates(files)
return
}
func getFile(file string, required bool) (files []string, err error) {
// Expand glob expression
expanded, err := filepath.Glob(file)
if err != nil {
return
}
if len(expanded) > 0 && expanded[0] != file {
for _, path := range expanded {
if globFiles, err := getFile(path, required); err != nil {
return files, err
} else {
files = append(files, globFiles...)
}
}
return misc.RemoveDuplicates(files), nil
}
fileInfo, err := os.Stat(file)
if err != nil {
if required {
return files, errors.New("getFile: File does not exist :" + file)
}
return files, nil
}
if fileInfo.IsDir() {
// Recurse over directory contents
err := filepath.Walk(file, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
if f.IsDir() {
return nil
}
newFiles, err := getFile(path, required)
if err != nil {
return err
}
files = append(files, newFiles...)
return nil
})
if err != nil {
return files, err
}
} else {
files = append(files, file)
// get dependencies for binaries
if _, err := elf.Open(file); err == nil {
if binaryDepFiles, err := getBinaryDeps(file); err != nil {
return files, err
} else {
files = append(files, binaryDepFiles...)
}
}
}
files = misc.RemoveDuplicates(files)
return
}
func getOskConfFontPath(oskConfPath string) (string, error) {
var path string
f, err := os.Open(oskConfPath)
if err != nil {
return path, err
}
defer f.Close()
s := bufio.NewScanner(f)
for s.Scan() {
fields := strings.Fields(s.Text())
// "key = val" is 3 fields
if len(fields) > 2 && fields[0] == "keyboard-font" {
path = fields[2]
}
}
if !exists(path) {
return path, errors.New("Unable to find font: " + path)
}
return path, nil
}
// Get a list of files and their dependencies related to supporting rootfs full
// disk (d)encryption
func getFdeFiles(devinfo deviceinfo.DeviceInfo) (files []string, err error) {
confFiles := []string{
"/etc/osk.conf",
"/etc/ts.conf",
"/etc/pointercal",
"/etc/fb.modes",
"/etc/directfbrc",
}
// TODO: this shouldn't be false? though some files (pointercal) don't always exist...
if files, err = getFiles(confFiles, false); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
}
// osk-sdl
oskFiles := []string{
"/usr/bin/osk-sdl",
"/sbin/cryptsetup",
"/usr/lib/libGL.so.1",
}
if filelist, err := getFiles(oskFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
files = append(files, filelist...)
}
fontFile, err := getOskConfFontPath("/etc/osk.conf")
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add file %q: %w", fontFile, err)
}
files = append(files, fontFile)
// Directfb
dfbFiles := []string{}
err = filepath.Walk("/usr/lib/directfb-1.7-7", func(path string, f os.FileInfo, err error) error {
if filepath.Ext(path) == ".so" {
dfbFiles = append(dfbFiles, path)
}
return nil
})
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add file %w", err)
}
if filelist, err := getFiles(dfbFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
files = append(files, filelist...)
}
// tslib
tslibFiles := []string{}
err = filepath.Walk("/usr/lib/ts", func(path string, f os.FileInfo, err error) error {
if filepath.Ext(path) == ".so" {
tslibFiles = append(tslibFiles, path)
}
return nil
})
if err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add file: %w", err)
}
libts, _ := filepath.Glob("/usr/lib/libts*")
tslibFiles = append(tslibFiles, libts...)
if filelist, err := getFiles(tslibFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
files = append(files, filelist...)
}
// mesa hw accel
if devinfo.MesaDriver != "" {
mesaFiles := []string{
"/usr/lib/libEGL.so.1",
"/usr/lib/libGLESv2.so.2",
"/usr/lib/libgbm.so.1",
"/usr/lib/libudev.so.1",
"/usr/lib/xorg/modules/dri/" + devinfo.MesaDriver + "_dri.so",
}
if filelist, err := getFiles(mesaFiles, true); err != nil {
return nil, fmt.Errorf("getFdeFiles: failed to add files: %w", err)
} else {
files = append(files, filelist...)
}
}
return
}
func getHookScripts() (files []string) {
scripts, _ := filepath.Glob("/etc/postmarketos-mkinitfs/hooks/*.sh")
files = append(files, scripts...)
return
}
func getInitfsExtraFiles(devinfo deviceinfo.DeviceInfo) (files []string, err error) {
log.Println("== Generating initramfs extra ==")
binariesExtra := []string{
"/lib/libz.so.1",
"/sbin/btrfs",
"/sbin/dmsetup",
"/sbin/e2fsck",
"/usr/sbin/parted",
"/usr/sbin/resize2fs",
"/usr/sbin/resize.f2fs",
}
log.Println("- Including extra binaries")
if filelist, err := getFiles(binariesExtra, true); err != nil {
return nil, err
} else {
files = append(files, filelist...)
}
// Hook files & scripts
if exists("/etc/postmarketos-mkinitfs/files-extra") {
log.Println("- Including hook files")
var hookFiles []string
hookFiles, err := getHookFiles("/etc/postmarketos-mkinitfs/files-extra")
if err != nil {
return nil, err
}
if filelist, err := getFiles(hookFiles, true); err != nil {
return nil, err
} else {
files = append(files, filelist...)
}
}
if exists("/usr/bin/osk-sdl") {
log.Println("- Including FDE support")
if fdeFiles, err := getFdeFiles(devinfo); err != nil {
return nil, err
} else {
files = append(files, fdeFiles...)
}
} else {
log.Println("- *NOT* including FDE support")
}
return
}
func getInitfsFiles(devinfo deviceinfo.DeviceInfo) (files []string, err error) {
log.Println("== Generating initramfs ==")
requiredFiles := []string{
"/bin/busybox",
"/bin/sh",
"/bin/busybox-extras",
"/usr/sbin/telnetd",
"/usr/sbin/kpartx",
"/etc/deviceinfo",
"/usr/bin/unudhcpd",
}
// Hook files & scripts
if exists("/etc/postmarketos-mkinitfs/files") {
log.Println("- Including hook files")
if hookFiles, err := getHookFiles("/etc/postmarketos-mkinitfs/files"); err != nil {
return nil, err
} else {
if filelist, err := getFiles(hookFiles, true); err != nil {
return nil, err
} else {
files = append(files, filelist...)
}
}
}
log.Println("- Including hook scripts")
hookScripts := getHookScripts()
files = append(files, hookScripts...)
log.Println("- Including required binaries")
if filelist, err := getFiles(requiredFiles, true); err != nil {
return nil, err
} else {
files = append(files, filelist...)
}
return
}
func getInitfsModules(devinfo deviceinfo.DeviceInfo, kernelVer string) (files []string, err error) {
log.Println("- Including kernel modules")
modDir := filepath.Join("/lib/modules", kernelVer)
if !exists(modDir) {
// dir /lib/modules/<kernel> if kernel built without module support, so just print a message
log.Printf("-- kernel module directory not found: %q, not including modules", modDir)
return
}
// modules.* required by modprobe
modprobeFiles, _ := filepath.Glob(filepath.Join(modDir, "modules.*"))
files = append(files, modprobeFiles...)
// module name (without extension), or directory (trailing slash is important! globs OK)
requiredModules := []string{
"loop",
"dm-crypt",
"kernel/fs/overlayfs/",
"kernel/crypto/",
"kernel/arch/*/crypto/",
}
for _, item := range requiredModules {
dir, file := filepath.Split(item)
if file == "" {
// item is a directory
dir = filepath.Join(modDir, dir)
dirs, _ := filepath.Glob(dir)
for _, d := range dirs {
if filelist, err := getModulesInDir(d); err != nil {
return nil, fmt.Errorf("getInitfsModules: unable to get modules dir %q: %w", d, err)
} else {
files = append(files, filelist...)
}
}
} else if dir == "" {
// item is a module name
if filelist, err := getModule(file, modDir); err != nil {
return nil, fmt.Errorf("getInitfsModules: unable to get module %q: %w", file, err)
} else {
files = append(files, filelist...)
}
} else {
log.Printf("Unknown module entry: %q", item)
}
}
// deviceinfo modules
for _, module := range strings.Fields(devinfo.ModulesInitfs) {
if filelist, err := getModule(module, modDir); err != nil {
return nil, fmt.Errorf("getInitfsModules: unable to get modules from deviceinfo: %w", err)
} else {
files = append(files, filelist...)
}
}
// /etc/postmarketos-mkinitfs/modules/*.modules
initfsModFiles, _ := filepath.Glob("/etc/postmarketos-mkinitfs/modules/*.modules")
for _, modFile := range initfsModFiles {
f, err := os.Open(modFile)
if err != nil {
return nil, fmt.Errorf("getInitfsModules: unable to open mkinitfs modules file %q: %w", modFile, err)
}
defer f.Close()
s := bufio.NewScanner(f)
for s.Scan() {
if filelist, err := getModule(s.Text(), modDir); err != nil {
return nil, fmt.Errorf("getInitfsModules: unable to get module file %q: %w", s.Text(), err)
} else {
files = append(files, filelist...)
}
}
}
return
}
func getKernelReleaseFile() (string, error) {
files, _ := filepath.Glob("/usr/share/kernel/*/kernel.release")
// only one kernel flavor supported
if len(files) != 1 {
return "", fmt.Errorf("only one kernel release/flavor is supported, found: %q", files)
}
return files[0], nil
}
func getKernelVersion() (string, error) {
var version string
releaseFile, err := getKernelReleaseFile()
if err != nil {
return version, err
}
contents, err := os.ReadFile(releaseFile)
if err != nil {
return version, err
}
return strings.TrimSpace(string(contents)), nil
}
func Copy(srcFile, dstFile string) error {
out, err := os.Create(dstFile)
if err != nil {
return err
}
defer out.Close()
in, err := os.Open(srcFile)
if err != nil {
return err
}
defer in.Close()
_, err = io.Copy(out, in)
if err != nil {
return err
}
return nil
}
func copyUbootFiles(path string, devinfo deviceinfo.DeviceInfo) error {
if devinfo.UbootBoardname == "" {
return nil
}
srcDir := filepath.Join("/usr/share/u-boot", devinfo.UbootBoardname)
entries, err := os.ReadDir(srcDir)
if err != nil {
return err
}
for _, entry := range entries {
sourcePath := filepath.Join(srcDir, entry.Name())
destPath := filepath.Join(path, entry.Name())
if err := Copy(sourcePath, destPath); err != nil {
return err
}
}
return nil
}
func generateInitfs(name string, path string, kernVer string, devinfo deviceinfo.DeviceInfo) error {
initfsArchive, err := archive.New()
if err != nil {
return err
}
requiredDirs := []string{
"/bin", "/sbin", "/usr/bin", "/usr/sbin", "/proc", "/sys",
"/dev", "/tmp", "/lib", "/boot", "/sysroot", "/etc",
}
for _, dir := range requiredDirs {
if err := initfsArchive.AddItem(dir, dir); err != nil {
return err
}
}
if files, err := getInitfsFiles(devinfo); err != nil {
return err
} else {
items := make(map[string]string)
// copy files into a map, where the source(key) and dest(value) are the
// same
for _, f := range files {
items[f] = f
}
if err := initfsArchive.AddItems(items); err != nil {
return err
}
}
if files, err := getInitfsModules(devinfo, kernVer); err != nil {
return err
} else {
items := make(map[string]string)
// copy files into a map, where the source(key) and dest(value) are the
// same
for _, f := range files {
items[f] = f
}
if err := initfsArchive.AddItems(items); err != nil {
return err
}
}
if err := initfsArchive.AddItem("/usr/share/postmarketos-mkinitfs/init.sh", "/init"); err != nil {
return err
}
// splash images
log.Println("- Including splash images")
splashFiles, _ := filepath.Glob("/usr/share/postmarketos-splashes/*.ppm.gz")
for _, file := range splashFiles {
// splash images are expected at /<file>
if err := initfsArchive.AddItem(file, filepath.Join("/", filepath.Base(file))); err != nil {
return err
}
}
// initfs_functions
if err := initfsArchive.AddItem("/usr/share/postmarketos-mkinitfs/init_functions.sh", "/init_functions.sh"); err != nil {
return err
}
log.Println("- Writing and verifying initramfs archive")
if err := initfsArchive.Write(filepath.Join(path, name), os.FileMode(0644)); err != nil {
return err
}
return nil
}
func generateInitfsExtra(name string, path string, devinfo deviceinfo.DeviceInfo) error {
initfsExtraArchive, err := archive.New()
if err != nil {
return err
}
if files, err := getInitfsExtraFiles(devinfo); err != nil {
return err
} else {
items := make(map[string]string)
// copy files into a map, where the source(key) and dest(value) are the
// same
for _, f := range files {
items[f] = f
}
if err := initfsExtraArchive.AddItems(items); err != nil {
return err
}
}
log.Println("- Writing and verifying initramfs-extra archive")
if err := initfsExtraArchive.Write(filepath.Join(path, name), os.FileMode(0644)); err != nil {
return err
}
return nil
}
func stripExts(file string) string {
return strings.Split(file, ".")[0]
}
func getModulesInDir(modPath string) (files []string, err error) {
err = filepath.Walk(modPath, func(path string, f os.FileInfo, err error) error {
// TODO: need to support more extensions?
if filepath.Ext(path) != ".ko" && filepath.Ext(path) != ".xz" {
return nil
}
files = append(files, path)
return nil
})
if err != nil {
return nil, err
}
return
}
// Given a module name, e.g. 'dwc_wdt', resolve the full path to the module
// file and all of its dependencies.
// Note: it's not necessarily fatal if the module is not found, since it may
// have been built into the kernel
// TODO: look for it in modules.builtin, and make it fatal if it can't be found
// anywhere
func getModule(modName string, modDir string) (files []string, err error) {
modDep := filepath.Join(modDir, "modules.dep")
if !exists(modDep) {
return nil, fmt.Errorf("kernel module.dep not found: %s", modDir)
}
fd, err := os.Open(modDep)
if err != nil {
return nil, fmt.Errorf("unable to open modules.dep: %w", err)
}
defer fd.Close()
deps, err := getModuleDeps(modName, fd)
if err != nil {
return nil, err
}
for _, dep := range deps {
p := filepath.Join(modDir, dep)
if !exists(p) {
return nil, fmt.Errorf("tried to include a module that doesn't exist in the modules directory (%s): %s", modDir, p)
}
files = append(files, p)
}
return
}
// Get the canonicalized name for the module as represented in the given modules.dep io.reader
func getModuleDeps(modName string, modulesDep io.Reader) ([]string, error) {
var deps []string
// split the module name on - and/or _, build a regex for matching
splitRe := regexp.MustCompile("[-_]+")
modNameReStr := splitRe.ReplaceAllString(modName, "[-_]+")
re := regexp.MustCompile("^" + modNameReStr + "$")
s := bufio.NewScanner(modulesDep)
for s.Scan() {
fields := strings.Fields(s.Text())
if len(fields) == 0 {
continue
}
fields[0] = strings.TrimSuffix(fields[0], ":")
found := re.FindAll([]byte(filepath.Base(stripExts(fields[0]))), -1)
if len(found) > 0 {
deps = append(deps, fields...)
break
}
}
if err := s.Err(); err != nil {
log.Print("Unable to get module + dependencies: ", modName)
return deps, err
}
return deps, nil
}

View File

@@ -14,31 +14,11 @@ import (
)
type DeviceInfo struct {
AppendDtb string
Arch string
UbootBoardname string
BootimgAppendSEAndroidEnforce string
BootimgBlobpack string
BootimgDtbSecond string
BootimgMtkMkimage string
BootimgPxa string
BootimgQcdt string
Dtb string
FlashKernelOnUpdate string
FlashOffsetBase string
FlashOffsetKernel string
FlashOffsetRamdisk string
FlashOffsetSecond string
FlashOffsetTags string
FlashPagesize string
GenerateBootimg string
GenerateLegacyUbootInitfs string
InitfsCompression string
KernelCmdline string
LegacyUbootLoadAddress string
MesaDriver string
MkinitfsPostprocess string
ModulesInitfs string
InitfsCompression string
InitfsExtraCompression string
MesaDriver string
ModulesInitfs string
UbootBoardname string
}
func ReadDeviceinfo(file string) (DeviceInfo, error) {

View File

@@ -21,6 +21,7 @@ func TestNameToField(t *testing.T) {
{"deviceinfo_modules_initfs", "ModulesInitfs"},
{"modules_initfs", "ModulesInitfs"},
{"deviceinfo_modules_initfs___", "ModulesInitfs"},
{"deviceinfo_initfs_extra_compression", "InitfsExtraCompression"},
}
for _, table := range tables {
@@ -43,15 +44,9 @@ func TestUnmarshal(t *testing.T) {
{"ModulesInitfs", "deviceinfo_modules_initfs=\"panfrost foo bar bazz\"\n", "panfrost foo bar bazz"},
{"ModulesInitfs", "deviceinfo_modules_initfs=\"panfrost foo bar bazz\"", "panfrost foo bar bazz"},
// line with multiple '='
{"KernelCmdline",
"deviceinfo_kernel_cmdline=\"PMOS_NO_OUTPUT_REDIRECT fw_devlink=off nvme_core.default_ps_max_latency_us=5500 pcie_aspm.policy=performance\"\n",
"PMOS_NO_OUTPUT_REDIRECT fw_devlink=off nvme_core.default_ps_max_latency_us=5500 pcie_aspm.policy=performance"},
{"InitfsCompression", "deviceinfo_initfs_compression=zstd:--foo=1 -T0 --bar=bazz", "zstd:--foo=1 -T0 --bar=bazz"},
// empty option
{"ModulesInitfs", "deviceinfo_modules_initfs=\"\"\n", ""},
{"Dtb", "deviceinfo_dtb=\"freescale/imx8mq-librem5-r2 freescale/imx8mq-librem5-r3 freescale/imx8mq-librem5-r4\"\n",
"freescale/imx8mq-librem5-r2 freescale/imx8mq-librem5-r3 freescale/imx8mq-librem5-r4"},
// valid deviceinfo line, just not used in this module
{"", "deviceinfo_codename=\"pine64-pinebookpro\"", ""},
// line with comment at the end
{"MesaDriver", "deviceinfo_mesa_driver=\"panfrost\" # this is a nice driver", "panfrost"},
{"", "# this is a comment!\n", ""},