smallsolar
/
kiwix-build
mirror of https://github.com/kiwix/kiwix-build.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #142 from kiwix/split_files 

Split files
This commit is contained in:
Matthieu Gautier 2018-04-04 22:17:41 +02:00 committed by GitHub
parent 1513204f42 72002c8f34
commit 70a6fa3860
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
39 changed files with 507 additions and 407 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
MANIFEST.in Normal file
View File

@ -0,0 +1,4 @@
include kiwixbuild/templates/cmake_cross_file.txt
include kiwixbuild/templates/meson_cross_file.txt
include kiwixbuild/templates/cmake_android_cross_file.txt
recursise-include kiwixbuild/patches *.patch

71
README.md
View File

@ -20,58 +20,62 @@ Before anything else you need to install Python3 related tools. On Debian
based systems: based systems:
``` ```
sudo apt-get install python3-pip virtualenv $ sudo apt-get install python3-pip virtualenv
``` ```
Then install Meson: Create a virtual environment to install python module in it instead
of modifying the system.
``` ```
virtualenv -p python3 ./ # Create virtualenv $ virtualenv -p python3 ./ # Create virtualenv
source bin/activate # Activate the virtualenv $ source bin/activate # Activate the virtualenv
pip3 install meson # Install Meson
hash -r # Refresh bash paths
``` ```
Finally we need the Ninja tool (available in the $PATH). Here is a Then, download and install kiwix-build and its dependencies:
solution to download, build and install it locally: ```
$ git clone git://github.com/kiwix/kiwix-build.git
$ cd kiwix-build
$ pip install .
$ hash -r # Refresh bash paths
```
If your distribution doesn't provide ninja version > 1.6 you can get it
this way :
``` ```
git clone git://github.com/ninja-build/ninja.git $ wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
cd ninja $ unzip ninja-linux.zip ninja -d $HOME/bin
git checkout release
./configure.py --bootstrap
mkdir ../bin
cp ninja ../bin
cd ..
``` ```
# Compilation # Compilation
The compilation is handled by `kiwix-build.py`. It will compile The compilation is handled by the `kiwix-build` command. It will compile
everything. If you are using a supported platform (Redhat or Debian everything. If you are using a supported platform (Redhat or Debian
based) it will install missing packages using `sudo`. You can get based) it will install missing packages using `sudo`. You can get
`kiwix-build.py` usage like this: `kiwix-build` usage like this:
``` ```
./kiwix-build.py -h $ kiwix-build --help
``` ```
## Target ## Target
By default, `kiwix-build.py` will build `kiwix-tools`. If you want to You may want to compile a specific target so you will have to specify it on the
compile another target only (let's said kiwixlib), you can specify it: command line :
``` ```
./kiwix-build Kiwixlib $ kiwix-build kiwix-lib # will build kiwix-build and its dependencies
$ kiwix-build zim-tools # will build zim-tools and its dependencies
``` ```
By default, `kiwix-build` will build `kiwix-tools` .
## Target platform ## Target platform
By default, `kiwix-build.py` will build everything for the current (native) By default, `kiwix-build` will build everything for the current (native)
platform using dynamic linkage (hence the `native_dyn` of the platform using dynamic linkage (`native_dyn`).
BUILD_native_dyn directory).
But you can select another target platform using the option But you can select another target platform using the option
`target-platform`. For now, there is ten different supported `--target-platform`. For now, there is ten different supported
platforms : platforms :
- native_dyn - native_dyn
@ -85,10 +89,15 @@ platforms :
- android_x86 - android_x86
- android_x86_64 - android_x86_64
So, if you want to compile for win32 using static linkage: So, if you want to compile `kiwix-tools` for win32 using static linkage:
``` ```
./kiwix-build.py --target-platform win32_dyn $ kiwix-build --target-platform win32_dyn
```
Or, android apk for android_arm :
```
$ kiwix-build --target-platform android_arm kiwix-android
``` ```
# Outputs # Outputs
@ -96,9 +105,9 @@ So, if you want to compile for win32 using static linkage:
Kiwix-build.py will create several directories: Kiwix-build.py will create several directories:
- `ARCHIVES`: All the downloaded archives go there. - `ARCHIVES`: All the downloaded archives go there.
- `SOURCES`: All the sources (extracted from archives and patched) go there. - `SOURCES`: All the sources (extracted from archives and patched) go there.
- `BUILD_native_dyn`: All the build files go there. - `BUILD_<target_platform>`: All the build files go there.
- `BUILD_native_dyn/INSTALL`: The installed files go there. - `BUILD_<target_platform>/INSTALL`: The installed files go there.
- `BUILD_native_dyn/LOGS`: The logs files of the build. - `BUILD_<target_platform>/LOGS`: The logs files of the build.
If you want to install all those directories elsewhere, you can pass the If you want to install all those directories elsewhere, you can pass the
`--working-dir` option to `kiwix-build.py`: `--working-dir` option to `kiwix-build`:

369
kiwix-build.py → kiwixbuild/__init__.py Executable file → Normal file
View File

@ -8,9 +8,9 @@ import subprocess
import platform import platform
from collections import OrderedDict from collections import OrderedDict
from dependencies import Dependency from .toolchains import Toolchain
from dependency_utils import ReleaseDownload, Builder, GitClone from .dependencies import Dependency
from utils import ( from .utils import (
pj, pj,
remove_duplicates, remove_duplicates,
add_execution_right, add_execution_right,
@ -47,7 +47,7 @@ PACKAGE_NAME_MAPPERS = {
'gumbo' : ['gumbo-parser-devel'], 'gumbo' : ['gumbo-parser-devel'],
}, },
'fedora_native_static': { 'fedora_native_static': {
'COMMON': _fedora_common + ['glibc-static', 'libstdc++-static'], 'COMMON': _fedora_common,
'zlib': ['zlib-devel', 'zlib-static'], 'zlib': ['zlib-devel', 'zlib-static'],
'lzma': ['xz-devel', 'xz-static'] 'lzma': ['xz-devel', 'xz-static']
# Either there is no packages, or no static or too old # Either there is no packages, or no static or too old
@ -118,12 +118,6 @@ PACKAGE_NAME_MAPPERS = {
} }
def which(name):
command = "which {}".format(name)
output = subprocess.check_output(command, shell=True)
return output[:-1].decode()
class TargetInfo: class TargetInfo:
def __init__(self, build, static, toolchains, hosts=None): def __init__(self, build, static, toolchains, hosts=None):
self.build = build self.build = build
@ -536,355 +530,6 @@ class BuildEnv:
with open(autoskip_file, 'w'): with open(autoskip_file, 'w'):
pass pass
class _MetaToolchain(type):
def __new__(cls, name, bases, dct):
_class = type.__new__(cls, name, bases, dct)
if name != 'Toolchain':
Toolchain.all_toolchains[name] = _class
return _class
class Toolchain(metaclass=_MetaToolchain):
all_toolchains = {}
configure_option = ""
cmake_option = ""
exec_wrapper_def = ""
Builder = None
Source = None
def __init__(self, buildEnv):
self.buildEnv = buildEnv
self.source = self.Source(self) if self.Source else None
self.builder = self.Builder(self) if self.Builder else None
@property
def full_name(self):
return "{name}-{version}".format(
name = self.name,
version = self.version)
@property
def source_path(self):
return pj(self.buildEnv.source_dir, self.source.source_dir)
@property
def _log_dir(self):
return self.buildEnv.log_dir
def set_env(self, env):
pass
def set_compiler(self, env):
pass
def command(self, name, function, *args):
print(" {} {} : ".format(name, self.name), end="", flush=True)
log = pj(self._log_dir, 'cmd_{}_{}.log'.format(name, self.name))
context = Context(name, log, True)
try:
ret = function(*args, context=context)
context._finalise()
print("OK")
return ret
except SkipCommand:
print("SKIP")
except subprocess.CalledProcessError:
print("ERROR")
try:
with open(log, 'r') as f:
print(f.read())
except:
pass
raise StopBuild()
except:
print("ERROR")
raise
class mingw32_toolchain(Toolchain):
name = 'mingw32'
arch_full = 'i686-w64-mingw32'
@property
def root_path(self):
return self.buildEnv.cross_config['root_path']
@property
def binaries(self):
return {k:which('{}-{}'.format(self.arch_full, v))
for k, v in (('CC', 'gcc'),
('CXX', 'g++'),
('AR', 'ar'),
('STRIP', 'strip'),
('WINDRES', 'windres'),
('RANLIB', 'ranlib'))
}
@property
def exec_wrapper_def(self):
try:
which('wine')
except subprocess.CalledProcessError:
return ""
else:
return "exec_wrapper = 'wine'"
@property
def configure_option(self):
return '--host={}'.format(self.arch_full)
def get_bin_dir(self):
return [pj(self.root_path, 'bin')]
def set_env(self, env):
env['PKG_CONFIG_LIBDIR'] = pj(self.root_path, 'lib', 'pkgconfig')
env['LIBS'] = " ".join(self.buildEnv.cross_config['extra_libs']) + " " +env['LIBS']
def set_compiler(self, env):
for k, v in self.binaries.items():
env[k] = v
class android_ndk(Toolchain):
name = 'android-ndk'
version = 'r13b'
gccver = '4.9.x'
@property
def api(self):
return '21' if self.arch in ('arm64', 'mips64', 'x86_64') else '14'
@property
def platform(self):
return 'android-'+self.api
@property
def arch(self):
return self.buildEnv.platform_info.arch
@property
def arch_full(self):
return self.buildEnv.platform_info.arch_full
@property
def toolchain(self):
return self.arch_full+"-4.9"
@property
def root_path(self):
return pj(self.builder.install_path, 'sysroot')
@property
def binaries(self):
binaries = ((k,'{}-{}'.format(self.arch_full, v))
for k, v in (('CC', 'gcc'),
('CXX', 'g++'),
('AR', 'ar'),
('STRIP', 'strip'),
('WINDRES', 'windres'),
('RANLIB', 'ranlib'),
('LD', 'ld'))
)
return {k:pj(self.builder.install_path, 'bin', v)
for k,v in binaries}
@property
def configure_option(self):
return '--host={}'.format(self.arch_full)
@property
def full_name(self):
return "{name}-{version}-{arch}-{api}".format(
name = self.name,
version = self.version,
arch = self.arch,
api = self.api)
class Source(ReleaseDownload):
archive = Remotefile('android-ndk-r13b-linux-x86_64.zip',
'3524d7f8fca6dc0d8e7073a7ab7f76888780a22841a6641927123146c3ffd29c',
'https://dl.google.com/android/repository/android-ndk-r13b-linux-x86_64.zip')
@property
def source_dir(self):
return "{}-{}".format(
self.target.name,
self.target.version)
class Builder(Builder):
@property
def install_path(self):
return self.build_path
def _build_platform(self, context):
context.try_skip(self.build_path)
script = pj(self.source_path, 'build/tools/make_standalone_toolchain.py')
add_execution_right(script)
command = '{script} --arch={arch} --api={api} --install-dir={install_dir} --force'
command = command.format(
script=script,
arch=self.target.arch,
api=self.target.api,
install_dir=self.install_path
)
self.buildEnv.run_command(command, self.build_path, context)
def _fix_permission_right(self, context):
context.try_skip(self.build_path)
bin_dirs = [pj(self.install_path, 'bin'),
pj(self.install_path, self.target.arch_full, 'bin'),
pj(self.install_path, 'libexec', 'gcc', self.target.arch_full, self.target.gccver)
]
for root, dirs, files in os.walk(self.install_path):
if not root in bin_dirs:
continue
for file_ in files:
file_path = pj(root, file_)
if os.path.islink(file_path):
continue
add_execution_right(file_path)
def build(self):
self.command('build_platform', self._build_platform)
self.command('fix_permission_right', self._fix_permission_right)
def get_bin_dir(self):
return [pj(self.builder.install_path, 'bin')]
def set_env(self, env):
env['PKG_CONFIG_LIBDIR'] = pj(self.root_path, 'lib', 'pkgconfig')
env['CFLAGS'] = '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot={} '.format(self.root_path) + env['CFLAGS']
env['CXXFLAGS'] = '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot={} '.format(self.root_path) + env['CXXFLAGS']
env['LDFLAGS'] = '--sysroot={} '.format(self.root_path) + env['LDFLAGS']
#env['CFLAGS'] = ' -fPIC -D_FILE_OFFSET_BITS=64 -O3 '+env['CFLAGS']
#env['CXXFLAGS'] = (' -D__OPTIMIZE__ -fno-strict-aliasing '
# ' -DU_HAVE_NL_LANGINFO_CODESET=0 '
# '-DU_STATIC_IMPLEMENTATION -O3 '
# '-DU_HAVE_STD_STRING -DU_TIMEZONE=0 ')+env['CXXFLAGS']
env['NDK_DEBUG'] = '0'
def set_compiler(self, env):
env['CC'] = self.binaries['CC']
env['CXX'] = self.binaries['CXX']
class android_sdk(Toolchain):
name = 'android-sdk'
version = 'r25.2.3'
class Source(ReleaseDownload):
archive = Remotefile('tools_r25.2.3-linux.zip',
'1b35bcb94e9a686dff6460c8bca903aa0281c6696001067f34ec00093145b560',
'https://dl.google.com/android/repository/tools_r25.2.3-linux.zip')
class Builder(Builder):
@property
def install_path(self):
return pj(self.buildEnv.toolchain_dir, self.target.full_name)
def _build_platform(self, context):
context.try_skip(self.install_path)
tools_dir = pj(self.install_path, 'tools')
shutil.copytree(self.source_path, tools_dir)
script = pj(tools_dir, 'android')
command = '{script} --verbose update sdk -a --no-ui --filter {packages}'
command = command.format(
script=script,
packages = ','.join(str(i) for i in [1,2,8,34,162])
)
# packages correspond to :
# - 1 : Android SDK Tools, revision 25.2.5
# - 2 : Android SDK Platform-tools, revision 25.0.3
# - 8 : Android SDK Build-tools, revision 24.0.1
# - 34 : SDK Platform Android 7.0, API 24, revision 2
# - 162 : Android Support Repository, revision 44
self.buildEnv.run_command(command, self.install_path, context, input="y\n")
def _fix_licenses(self, context):
context.try_skip(self.install_path)
os.makedirs(pj(self.install_path, 'licenses'), exist_ok=True)
with open(pj(self.install_path, 'licenses', 'android-sdk-license'), 'w') as f:
f.write("\n8933bad161af4178b1185d1a37fbf41ea5269c55\nd56f5187479451eabf01fb78af6dfcb131a6481e")
def build(self):
self.command('build_platform', self._build_platform)
self.command('fix_licenses', self._fix_licenses)
def get_bin_dir(self):
return []
def set_env(self, env):
env['ANDROID_HOME'] = self.builder.install_path
class armhf_toolchain(Toolchain):
name = 'armhf'
arch_full = 'arm-linux-gnueabihf'
class Source(GitClone):
git_remote = "https://github.com/raspberrypi/tools"
git_dir = "raspberrypi-tools"
@property
def root_path(self):
return pj(self.source_path, 'arm-bcm2708', 'gcc-linaro-arm-linux-gnueabihf-raspbian-x64')
@property
def binaries(self):
binaries = ((k,'{}-{}'.format(self.arch_full, v))
for k, v in (('CC', 'gcc'),
('CXX', 'g++'),
('AR', 'ar'),
('STRIP', 'strip'),
('WINDRES', 'windres'),
('RANLIB', 'ranlib'),
('LD', 'ld'))
)
return {k:pj(self.root_path, 'bin', v)
for k,v in binaries}
@property
def exec_wrapper_def(self):
try:
which('qemu-arm')
except subprocess.CalledProcessError:
return ""
else:
return "exec_wrapper = 'qemu-arm'"
@property
def configure_option(self):
return '--host={}'.format(self.arch_full)
def get_bin_dir(self):
return [pj(self.root_path, 'bin')]
def set_env(self, env):
env['PKG_CONFIG_LIBDIR'] = pj(self.root_path, 'lib', 'pkgconfig')
env['CFLAGS'] = " -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 "+env['CFLAGS']
env['CXXFLAGS'] = " -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 "+env['CXXFLAGS']
env['LIBS'] = " ".join(self.buildEnv.cross_config['extra_libs']) + " " +env['LIBS']
env['QEMU_LD_PREFIX'] = pj(self.root_path, "arm-linux-gnueabihf", "libc")
env['QEMU_SET_ENV'] = "LD_LIBRARY_PATH={}".format(
':'.join([
pj(self.root_path, "arm-linux-gnueabihf", "lib"),
pj(self.buildEnv.install_dir, 'lib'),
pj(self.buildEnv.install_dir, self.buildEnv.libprefix)
]))
def set_compiler(self, env):
env['CC'] = self.binaries['CC']
env['CXX'] = self.binaries['CXX']
class iOS_sdk(Toolchain):
pass
class Builder: class Builder:
def __init__(self, options): def __init__(self, options):
@ -897,7 +542,7 @@ class Builder:
self.add_targets(targetDef, _targets) self.add_targets(targetDef, _targets)
dependencies = self.order_dependencies(_targets, targetDef) dependencies = self.order_dependencies(_targets, targetDef)
dependencies = list(remove_duplicates(dependencies)) dependencies = list(remove_duplicates(dependencies))
for dep in dependencies: for dep in dependencies:
if self.options.build_deps_only and dep == targetDef: if self.options.build_deps_only and dep == targetDef:
continue continue
@ -1021,10 +666,10 @@ def parse_args():
sys.exit(1) sys.exit(1)
return options return options
def main():
if __name__ == "__main__":
options = parse_args() options = parse_args()
options.working_dir = os.path.abspath(options.working_dir) options.working_dir = os.path.abspath(options.working_dir)
setup_print_progress(options.show_progress) setup_print_progress(options.show_progress)
builder = Builder(options) builder = Builder(options)
builder.run() builder.run()

7
kiwixbuild/__main__.py Normal file
View File

@ -0,0 +1,7 @@
#!/usr/bin/env python3
from . import main
if __name__ == "__main__":
main()

7
dependencies.py → kiwixbuild/dependencies.py
View File

@ -1,7 +1,7 @@
import shutil, os, json import shutil, os, json
from urllib.parse import urlparse from urllib.parse import urlparse
from dependency_utils import ( from kiwixbuild.dependency_utils import (
Dependency, Dependency,
ReleaseDownload, ReleaseDownload,
GitClone, GitClone,
@ -14,7 +14,7 @@ from dependency_utils import (
NoopBuilder, NoopBuilder,
Builder as BaseBuilder) Builder as BaseBuilder)
from utils import Remotefile, pj, SkipCommand, copy_tree, add_execution_right from kiwixbuild.utils import Remotefile, pj, SkipCommand, copy_tree, add_execution_right
# ************************************* # *************************************
# Missing dependencies # Missing dependencies
@ -22,7 +22,6 @@ from utils import Remotefile, pj, SkipCommand, copy_tree, add_execution_right
# exist in your "distri" (linux/mac) ? # exist in your "distri" (linux/mac) ?
# If not, we need to compile them here # If not, we need to compile them here
# ************************************* # *************************************
# aria2
# Argtable # Argtable
# MSVirtual # MSVirtual
# Android # Android
@ -415,7 +414,7 @@ class AllBaseDependencies(Dependency):
@property @property
def dependencies(self): def dependencies(self):
base_deps = ['zlib', 'lzma', 'xapian-core', 'gumbo', 'pugixml', 'libmicrohttpd', 'libaria2'] base_deps = ['zlib', 'lzma', 'xapian-core', 'gumbo', 'pugixml', 'libmicrohttpd']
if self.buildEnv.platform_info.build != 'native': if self.buildEnv.platform_info.build != 'native':
base_deps += ["icu4c_cross-compile"] base_deps += ["icu4c_cross-compile"]
else: else:

4
dependency_utils.py → kiwixbuild/dependency_utils.py
View File

@ -2,8 +2,8 @@ import subprocess
import os import os
import shutil import shutil
from utils import pj, Context, SkipCommand, extract_archive, Defaultdict, StopBuild from kiwixbuild.utils import pj, Context, SkipCommand, extract_archive, Defaultdict, StopBuild
from dependency_versions import main_project_versions, base_deps_versions from kiwixbuild.dependency_versions import main_project_versions, base_deps_versions
SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))

0
dependency_versions.py → kiwixbuild/dependency_versions.py
View File

0
patches/CTPP2SourceLoader.hpp.patch → kiwixbuild/patches/CTPP2SourceLoader.hpp.patch
View File

0
patches/ctpp2_compile_ctpp2c_static.patch → kiwixbuild/patches/ctpp2_compile_ctpp2c_static.patch
View File

0
patches/ctpp2_dll_export_VMExecutable.patch → kiwixbuild/patches/ctpp2_dll_export_VMExecutable.patch
View File

0
patches/ctpp2_fix-static-libname.patch → kiwixbuild/patches/ctpp2_fix-static-libname.patch
View File

0
patches/ctpp2_iconv_support.patch → kiwixbuild/patches/ctpp2_iconv_support.patch
View File

0
patches/ctpp2_include.patch → kiwixbuild/patches/ctpp2_include.patch
View File

0
patches/ctpp2_mingw32.patch → kiwixbuild/patches/ctpp2_mingw32.patch
View File

0
patches/ctpp2_no_src_modification.patch → kiwixbuild/patches/ctpp2_no_src_modification.patch
View File

0
patches/ctpp2_win_install_lib_in_lib_dir.patch → kiwixbuild/patches/ctpp2_win_install_lib_in_lib_dir.patch
View File

0
patches/icu4c_android_elf64_st_info.patch → kiwixbuild/patches/icu4c_android_elf64_st_info.patch
View File

0
patches/icu4c_custom_data.patch → kiwixbuild/patches/icu4c_custom_data.patch
View File

0
patches/icu4c_fix_static_lib_name_mingw.patch → kiwixbuild/patches/icu4c_fix_static_lib_name_mingw.patch
View File

0
patches/icu4c_noxlocale.patch → kiwixbuild/patches/icu4c_noxlocale.patch
View File

0
patches/libaria2_android.patch → kiwixbuild/patches/libaria2_android.patch
View File

0
patches/pugixml_meson.patch → kiwixbuild/patches/pugixml_meson.patch
View File

0
patches/zlib_std_libname.patch → kiwixbuild/patches/zlib_std_libname.patch
View File

0
templates/bash_wrapper.sh → kiwixbuild/templates/bash_wrapper.sh
View File

0
templates/cmake_android_cross_file.txt → kiwixbuild/templates/cmake_android_cross_file.txt
View File

0
templates/cmake_cross_file.txt → kiwixbuild/templates/cmake_cross_file.txt
View File

0
templates/meson_android_cross_file.txt → kiwixbuild/templates/meson_android_cross_file.txt
View File

0
templates/meson_cross_file.txt → kiwixbuild/templates/meson_cross_file.txt
View File

4
kiwixbuild/toolchains/__init__.py Normal file
View File

@ -0,0 +1,4 @@
from . import android_ndk, android_sdk, armhf, ios, mingw32
from .base_toolchain import Toolchain

133
kiwixbuild/toolchains/android_ndk.py Normal file
View File

@ -0,0 +1,133 @@
import os
from .base_toolchain import Toolchain
from kiwixbuild.dependency_utils import ReleaseDownload, Builder
from kiwixbuild.utils import Remotefile, add_execution_right
pj = os.path.join
class android_ndk(Toolchain):
name = 'android-ndk'
version = 'r13b'
gccver = '4.9.x'
@property
def api(self):
return '21' if self.arch in ('arm64', 'mips64', 'x86_64') else '14'
@property
def platform(self):
return 'android-'+self.api
@property
def arch(self):
return self.buildEnv.platform_info.arch
@property
def arch_full(self):
return self.buildEnv.platform_info.arch_full
@property
def toolchain(self):
return self.arch_full+"-4.9"
@property
def root_path(self):
return pj(self.builder.install_path, 'sysroot')
@property
def binaries(self):
binaries = ((k,'{}-{}'.format(self.arch_full, v))
for k, v in (('CC', 'gcc'),
('CXX', 'g++'),
('AR', 'ar'),
('STRIP', 'strip'),
('WINDRES', 'windres'),
('RANLIB', 'ranlib'),
('LD', 'ld'))
)
return {k:pj(self.builder.install_path, 'bin', v)
for k,v in binaries}
@property
def configure_option(self):
return '--host={}'.format(self.arch_full)
@property
def full_name(self):
return "{name}-{version}-{arch}-{api}".format(
name = self.name,
version = self.version,
arch = self.arch,
api = self.api)
class Source(ReleaseDownload):
archive = Remotefile('android-ndk-r13b-linux-x86_64.zip',
'3524d7f8fca6dc0d8e7073a7ab7f76888780a22841a6641927123146c3ffd29c',
'https://dl.google.com/android/repository/android-ndk-r13b-linux-x86_64.zip')
@property
def source_dir(self):
return "{}-{}".format(
self.target.name,
self.target.version)
class Builder(Builder):
@property
def install_path(self):
return self.build_path
def _build_platform(self, context):
context.try_skip(self.build_path)
script = pj(self.source_path, 'build/tools/make_standalone_toolchain.py')
add_execution_right(script)
command = '{script} --arch={arch} --api={api} --install-dir={install_dir} --force'
command = command.format(
script=script,
arch=self.target.arch,
api=self.target.api,
install_dir=self.install_path
)
self.buildEnv.run_command(command, self.build_path, context)
def _fix_permission_right(self, context):
context.try_skip(self.build_path)
bin_dirs = [pj(self.install_path, 'bin'),
pj(self.install_path, self.target.arch_full, 'bin'),
pj(self.install_path, 'libexec', 'gcc', self.target.arch_full, self.target.gccver)
]
for root, dirs, files in os.walk(self.install_path):
if not root in bin_dirs:
continue
for file_ in files:
file_path = pj(root, file_)
if os.path.islink(file_path):
continue
add_execution_right(file_path)
def build(self):
self.command('build_platform', self._build_platform)
self.command('fix_permission_right', self._fix_permission_right)
def get_bin_dir(self):
return [pj(self.builder.install_path, 'bin')]
def set_env(self, env):
env['PKG_CONFIG_LIBDIR'] = pj(self.root_path, 'lib', 'pkgconfig')
env['CFLAGS'] = '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot={} '.format(self.root_path) + env['CFLAGS']
env['CXXFLAGS'] = '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot={} '.format(self.root_path) + env['CXXFLAGS']
env['LDFLAGS'] = '--sysroot={} '.format(self.root_path) + env['LDFLAGS']
#env['CFLAGS'] = ' -fPIC -D_FILE_OFFSET_BITS=64 -O3 '+env['CFLAGS']
#env['CXXFLAGS'] = (' -D__OPTIMIZE__ -fno-strict-aliasing '
# ' -DU_HAVE_NL_LANGINFO_CODESET=0 '
# '-DU_STATIC_IMPLEMENTATION -O3 '
# '-DU_HAVE_STD_STRING -DU_TIMEZONE=0 ')+env['CXXFLAGS']
env['NDK_DEBUG'] = '0'
def set_compiler(self, env):
env['CC'] = self.binaries['CC']
env['CXX'] = self.binaries['CXX']

57
kiwixbuild/toolchains/android_sdk.py Normal file
View File

@ -0,0 +1,57 @@
import os
import shutil
from .base_toolchain import Toolchain
from kiwixbuild.dependency_utils import ReleaseDownload, Builder
from kiwixbuild.utils import Remotefile
pj = os.path.join
class android_sdk(Toolchain):
name = 'android-sdk'
version = 'r25.2.3'
class Source(ReleaseDownload):
archive = Remotefile('tools_r25.2.3-linux.zip',
'1b35bcb94e9a686dff6460c8bca903aa0281c6696001067f34ec00093145b560',
'https://dl.google.com/android/repository/tools_r25.2.3-linux.zip')
class Builder(Builder):
@property
def install_path(self):
return pj(self.buildEnv.toolchain_dir, self.target.full_name)
def _build_platform(self, context):
context.try_skip(self.install_path)
tools_dir = pj(self.install_path, 'tools')
shutil.copytree(self.source_path, tools_dir)
script = pj(tools_dir, 'android')
command = '{script} --verbose update sdk -a --no-ui --filter {packages}'
command = command.format(
script=script,
packages = ','.join(str(i) for i in [1,2,8,34,162])
)
# packages correspond to :
# - 1 : Android SDK Tools, revision 25.2.5
# - 2 : Android SDK Platform-tools, revision 25.0.3
# - 8 : Android SDK Build-tools, revision 24.0.1
# - 34 : SDK Platform Android 7.0, API 24, revision 2
# - 162 : Android Support Repository, revision 44
self.buildEnv.run_command(command, self.install_path, context, input="y\n")
def _fix_licenses(self, context):
context.try_skip(self.install_path)
os.makedirs(pj(self.install_path, 'licenses'), exist_ok=True)
with open(pj(self.install_path, 'licenses', 'android-sdk-license'), 'w') as f:
f.write("\n8933bad161af4178b1185d1a37fbf41ea5269c55\nd56f5187479451eabf01fb78af6dfcb131a6481e")
def build(self):
self.command('build_platform', self._build_platform)
self.command('fix_licenses', self._fix_licenses)
def get_bin_dir(self):
return []
def set_env(self, env):
env['ANDROID_HOME'] = self.builder.install_path

66
kiwixbuild/toolchains/armhf.py Normal file
View File

@ -0,0 +1,66 @@
import os
import subprocess
from .base_toolchain import Toolchain
from kiwixbuild.dependency_utils import GitClone
from kiwixbuild.utils import Remotefile, which
pj = os.path.join
class armhf_toolchain(Toolchain):
name = 'armhf'
arch_full = 'arm-linux-gnueabihf'
class Source(GitClone):
git_remote = "https://github.com/raspberrypi/tools"
git_dir = "raspberrypi-tools"
@property
def root_path(self):
return pj(self.source_path, 'arm-bcm2708', 'gcc-linaro-arm-linux-gnueabihf-raspbian-x64')
@property
def binaries(self):
binaries = ((k,'{}-{}'.format(self.arch_full, v))
for k, v in (('CC', 'gcc'),
('CXX', 'g++'),
('AR', 'ar'),
('STRIP', 'strip'),
('WINDRES', 'windres'),
('RANLIB', 'ranlib'),
('LD', 'ld'))
)
return {k:pj(self.root_path, 'bin', v)
for k,v in binaries}
@property
def exec_wrapper_def(self):
try:
which('qemu-arm')
except subprocess.CalledProcessError:
return ""
else:
return "exec_wrapper = 'qemu-arm'"
@property
def configure_option(self):
return '--host={}'.format(self.arch_full)
def get_bin_dir(self):
return [pj(self.root_path, 'bin')]
def set_env(self, env):
env['PKG_CONFIG_LIBDIR'] = pj(self.root_path, 'lib', 'pkgconfig')
env['CFLAGS'] = " -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 "+env['CFLAGS']
env['CXXFLAGS'] = " -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 "+env['CXXFLAGS']
env['LIBS'] = " ".join(self.buildEnv.cross_config['extra_libs']) + " " +env['LIBS']
env['QEMU_LD_PREFIX'] = pj(self.root_path, "arm-linux-gnueabihf", "libc")
env['QEMU_SET_ENV'] = "LD_LIBRARY_PATH={}".format(
':'.join([
pj(self.root_path, "arm-linux-gnueabihf", "lib"),
pj(self.buildEnv.install_dir, 'lib'),
pj(self.buildEnv.install_dir, self.buildEnv.libprefix)
]))
def set_compiler(self, env):
env['CC'] = self.binaries['CC']
env['CXX'] = self.binaries['CXX']

70
kiwixbuild/toolchains/base_toolchain.py Normal file
View File

@ -0,0 +1,70 @@
import os
import subprocess
pj = os.path.join
from kiwixbuild.utils import Context, SkipCommand, StopBuild
class _MetaToolchain(type):
def __new__(cls, name, bases, dct):
_class = type.__new__(cls, name, bases, dct)
if name != 'Toolchain':
Toolchain.all_toolchains[name] = _class
return _class
class Toolchain(metaclass=_MetaToolchain):
all_toolchains = {}
configure_option = ""
cmake_option = ""
exec_wrapper_def = ""
Builder = None
Source = None
def __init__(self, buildEnv):
self.buildEnv = buildEnv
self.source = self.Source(self) if self.Source else None
self.builder = self.Builder(self) if self.Builder else None
@property
def full_name(self):
return "{name}-{version}".format(
name = self.name,
version = self.version)
@property
def source_path(self):
return pj(self.buildEnv.source_dir, self.source.source_dir)
@property
def _log_dir(self):
return self.buildEnv.log_dir
def set_env(self, env):
pass
def set_compiler(self, env):
pass
def command(self, name, function, *args):
print(" {} {} : ".format(name, self.name), end="", flush=True)
log = pj(self._log_dir, 'cmd_{}_{}.log'.format(name, self.name))
context = Context(name, log, True)
try:
ret = function(*args, context=context)
context._finalise()
print("OK")
return ret
except SkipCommand:
print("SKIP")
except subprocess.CalledProcessError:
print("ERROR")
try:
with open(log, 'r') as f:
print(f.read())
except:
pass
raise StopBuild()
except:
print("ERROR")
raise

4
kiwixbuild/toolchains/ios.py Normal file
View File

@ -0,0 +1,4 @@
from .base_toolchain import Toolchain
class iOS_sdk(Toolchain):
pass

51
kiwixbuild/toolchains/mingw32.py Normal file
View File

@ -0,0 +1,51 @@
import os
import subprocess
from .base_toolchain import Toolchain
from kiwixbuild.utils import which
pj = os.path.join
class mingw32_toolchain(Toolchain):
name = 'mingw32'
arch_full = 'i686-w64-mingw32'
@property
def root_path(self):
return self.buildEnv.cross_config['root_path']
@property
def binaries(self):
return {k:which('{}-{}'.format(self.arch_full, v))
for k, v in (('CC', 'gcc'),
('CXX', 'g++'),
('AR', 'ar'),
('STRIP', 'strip'),
('WINDRES', 'windres'),
('RANLIB', 'ranlib'))
}
@property
def exec_wrapper_def(self):
try:
which('wine')
except subprocess.CalledProcessError:
return ""
else:
return "exec_wrapper = 'wine'"
@property
def configure_option(self):
return '--host={}'.format(self.arch_full)
def get_bin_dir(self):
return [pj(self.root_path, 'bin')]
def set_env(self, env):
env['PKG_CONFIG_LIBDIR'] = pj(self.root_path, 'lib', 'pkgconfig')
env['LIBS'] = " ".join(self.buildEnv.cross_config['extra_libs']) + " " +env['LIBS']
def set_compiler(self, env):
for k, v in self.binaries.items():
env[k] = v

9
utils.py → kiwixbuild/utils.py
View File

@ -6,6 +6,7 @@ import shutil
import os, stat, sys import os, stat, sys
import urllib import urllib
import ssl import ssl
import subprocess
from collections import namedtuple, defaultdict from collections import namedtuple, defaultdict
pj = os.path.join pj = os.path.join
@ -15,6 +16,12 @@ g_print_progress = True
REMOTE_PREFIX = 'http://download.kiwix.org/dev/' REMOTE_PREFIX = 'http://download.kiwix.org/dev/'
def which(name):
command = "which {}".format(name)
output = subprocess.check_output(command, shell=True)
return output[:-1].decode()
def setup_print_progress(print_progress): def setup_print_progress(print_progress):
global g_print_progress global g_print_progress
g_print_progress = print_progress g_print_progress = print_progress
@ -63,6 +70,7 @@ def add_execution_right(file_path):
current_permissions = stat.S_IMODE(os.lstat(file_path).st_mode) current_permissions = stat.S_IMODE(os.lstat(file_path).st_mode)
os.chmod(file_path, current_permissions | stat.S_IXUSR) os.chmod(file_path, current_permissions | stat.S_IXUSR)
def copy_tree(src, dst, post_copy_function=None): def copy_tree(src, dst, post_copy_function=None):
os.makedirs(dst, exist_ok=True) os.makedirs(dst, exist_ok=True)
for root, dirs, files in os.walk(src): for root, dirs, files in os.walk(src):
@ -152,6 +160,7 @@ class Context:
def extract_archive(archive_path, dest_dir, topdir=None, name=None): def extract_archive(archive_path, dest_dir, topdir=None, name=None):
is_zip_archive = archive_path.endswith('.zip') is_zip_archive = archive_path.endswith('.zip')
archive = None
try: try:
if is_zip_archive: if is_zip_archive:
archive = zipfile.ZipFile(archive_path) archive = zipfile.ZipFile(archive_path)

45
setup.py Normal file
View File

@ -0,0 +1,45 @@
from setuptools import setup, find_packages
from codecs import open # To use a consistent encoding
from os import path
here = path.abspath(path.dirname(__file__))
# Get the long description from the relevant file
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
# Doc: http://pythonhosted.org/setuptools/setuptools.html#including-data-files.
setup(
name='kiwix-build',
version="0.1",
description=('A tool to build kiwix and openzim projects.'),
long_description=long_description,
url='https://github.com/kiwix/kiwix-build/',
author='Matthieu Gautier',
author_email='mgautier@kymeria.fr',
license='GPLv3',
# See https://pypi.python.org/pypi?%3Aaction=list_classifiers
classifiers=[
'Development Status :: 3 - Alpha',
# Indicate who your project is intended for
'Intended Audience :: Developers',
'Topic :: Software Development :: Build Tools',
# Specify the Python versions you support here. In particular, ensure
# that you indicate whether you support Python 2, Python 3 or both.
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
],
packages=find_packages(),
include_package_data=True,
install_requires=[
'meson==0.43.0'
],
entry_points={
'console_scripts': [
'kiwix-build = kiwixbuild:main'
]
}
)

5
travis/compile_all.py
View File

@ -11,8 +11,7 @@ import re
from urllib.request import urlretrieve from urllib.request import urlretrieve
from urllib.error import URLError from urllib.error import URLError
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) from kiwixbuild import dependency_versions
import dependency_versions
PLATFORM = environ['PLATFORM'] PLATFORM = environ['PLATFORM']
@ -56,7 +55,7 @@ Generated at {date}
def run_kiwix_build(target, platform, build_deps_only=False, make_release=False, make_dist=False): def run_kiwix_build(target, platform, build_deps_only=False, make_release=False, make_dist=False):
command = [str(Path(environ['TRAVIS_BUILD_DIR'])/'kiwix-build.py')] command = ['kiwix-build']
command.append(target) command.append(target)
command.append('--hide-progress') command.append('--hide-progress')
command.extend(['--target-platform', platform]) command.extend(['--target-platform', platform])

8
travis/install_extra_deps.sh
View File

@ -2,15 +2,13 @@
set -e set -e
orig_dir=$(pwd)
pip3 install --user --upgrade pip wheel pip3 install --user --upgrade pip wheel
pip3 install --user pillow meson==0.43.0 pip3 install --user pillow
pip3 install --user .
# ninja # ninja
wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
unzip ninja-linux.zip ninja unzip ninja-linux.zip ninja
cp ninja $HOME/bin cp ninja $HOME/bin
cd $orig_dir