Compare commits

...

13 Commits
extm ... bmk

Author SHA1 Message Date
Simon Glass
8543f02fd2 buildman: Fix remaining pylint warnings in boards.py
- Add module-level disable for too-many-lines
- Suppress consider-using-with for NamedTemporaryFile (intentional)
- Change unused linenum to _ in parse_file()
- Suppress too-many-arguments for _check_board()
- Add missing param documentation for parse_extended()

Cover-letter:
buildman: Fix pylint warnings in board.py and boards.py
This series addresses pylint warnings in the buildman board modules,
achieving a 10/10 pylint score.

For board.py, unavoidable warnings (too-many-instance-attributes,
too-few-public-methods, too-many-arguments) are suppressed since the
Board class is a legitimate data container.

For boards.py, the series:
- Fixes line-too-long and missing return documentation warnings
- Refactors complex functions by extracting helper methods to reduce
  too-many-branches and too-many-locals warnings
- Adds missing parameter documentation
- Suppresses remaining unavoidable warnings
END

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
ec3f33ef13 buildman: Extract _check_board() from select_boards()
Move the board checking logic into a separate static method to reduce
complexity of select_boards()

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
d0d41be6b0 buildman: Extract helper functions from output_is_new()
Move the srcdir walk and output file scanning into module-level
functions _check_srcdir_is_current() and _check_output_is_current()
to reduce complexity.

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
5d6a456d86 buildman: Split _start_defconfig_scans() into two functions
Split the setup logic into _collect_defconfigs() for gathering defconfig
file paths and _start_scanners() for spawning the parallel scan
processes. This reduces complexity and eliminates the too-many-locals
warning.

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
149680d860 buildman: Extract F: and N: tag handling in parse_file() into methods
Move the handling of F: (file path) and N: (name pattern) tags into
separate class methods _handle_f_tag() and _handle_n_tag(), and target
database updates into _add_targets()

This reduces complexity of parse_file()

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
fae9c4cb2f buildman: Extract target checking from scan() into _check_targets()
Move the logic that verifies exactly one TARGET_xxx option is set into
a separate method to reduce complexity of scan()

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
65ee7f05cb buildman: Extract defconfig loading from scan() into _load_defconfig()
Move the logic that handles #include preprocessing and config loading
into a separate method to reduce complexity of scan()

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
ad6fd76a6a buildman: Extract tag parsing from parse() into _parse_tag()
Move the handling of tag lines (name:, desc:, fragment:, targets:) into
a separate method to reduce complexity of parse()

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
d6735b5cf1 buildman: Use specific exception type in _fixup_arch()
Change bare except to catch AttributeError specifically, which occurs
when the ARCH_RV32I symbol is not found and get() returns None

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
7250a9bc97 buildman: Extract arch fixups from scan() into separate function
Move the architecture fixup logic (aarch64 and riscv adjustments) into
a new _fixup_arch() method to reduce complexity of scan().

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
1680c7ba7f buildman: Fix pylint warnings in boards.py
Fix line-too-long warnings by wrapping long lines, and add missing
return-type documentation to various functions.

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
f973ae99a4 buildman: Suppress pylint warnings in board.py
The Board class is a data container that legitimately needs many
attributes and constructor arguments. Suppress the following warnings:

- too-many-instance-attributes
- too-few-public-methods
- too-many-arguments

Also add missing type annotations to the docstring for consistency.

Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
2026-01-03 13:30:53 -07:00
Simon Glass
b133ad61cc Merge branch 'extm' into 'master'
Malloc debugging and test/py improvements

See merge request u-boot/u-boot!357
2026-01-03 13:24:17 -07:00
2 changed files with 389 additions and 206 deletions

View File

@@ -4,21 +4,23 @@
"""A single board which can be selected and built""" """A single board which can be selected and built"""
# pylint: disable=too-many-instance-attributes,too-few-public-methods
class Board: class Board:
"""A particular board that we can build""" """A particular board that we can build"""
# pylint: disable=too-many-arguments
def __init__(self, status, arch, cpu, soc, vendor, board_name, target, def __init__(self, status, arch, cpu, soc, vendor, board_name, target,
cfg_name, extended=None, orig_target=None): cfg_name, extended=None, orig_target=None):
"""Create a new board type. """Create a new board type.
Args: Args:
status: define whether the board is 'Active' or 'Orphaned' status (str): Either 'Active' or 'Orphaned'
arch: Architecture name (e.g. arm) arch (str): Architecture name (e.g. arm)
cpu: Cpu name (e.g. arm1136) cpu (str): Cpu name (e.g. arm1136)
soc: Name of SOC, or '' if none (e.g. mx31) soc (str): Name of SOC, or '' if none (e.g. mx31)
vendor: Name of vendor (e.g. armltd) vendor (str): Name of vendor (e.g. armltd)
board_name: Name of board (e.g. integrator) board_name (str): Name of board (e.g. integrator)
target: Target name (use make <target>_defconfig to configure) target (str): Target name (use make <target>_defconfig to configure)
cfg_name: Config-file name (in includes/configs/) cfg_name (str): Config-file name (in includes/configs/)
extended (boards.Extended): Extended board, if this board is one extended (boards.Extended): Extended board, if this board is one
orig_target (str): Name of target this extended board is based on orig_target (str): Name of target this extended board is based on
""" """

View File

@@ -3,6 +3,8 @@
# Author: Simon Glass <sjg@chromium.org> # Author: Simon Glass <sjg@chromium.org>
# Author: Masahiro Yamada <yamada.m@jp.panasonic.com> # Author: Masahiro Yamada <yamada.m@jp.panasonic.com>
# pylint: disable=too-many-lines
"""Maintains a list of boards and allows them to be selected""" """Maintains a list of boards and allows them to be selected"""
from collections import OrderedDict, namedtuple from collections import OrderedDict, namedtuple
@@ -56,6 +58,50 @@ def try_remove(fname):
raise raise
def _check_srcdir_is_current(ctime, srcdir):
"""Check if any Kconfig or MAINTAINERS files are newer than ctime
Args:
ctime (float): Reference time to compare against
srcdir (str): Directory containing Kconfig and MAINTAINERS files
Returns:
bool: True if all files are older than ctime
"""
for (dirpath, _, filenames) in os.walk(srcdir):
for filename in filenames:
if (fnmatch.fnmatch(filename, '*~') or
not fnmatch.fnmatch(filename, 'Kconfig*') and
not filename == 'MAINTAINERS'):
continue
filepath = os.path.join(dirpath, filename)
if ctime < os.path.getctime(filepath):
return False
return True
def _check_output_is_current(output, config_dir):
"""Check if output references any removed boards
Args:
output (str): Path to the output file
config_dir (str): Directory containing defconfig files
Returns:
bool: True if all referenced boards still exist
"""
with open(output, encoding="utf-8") as inf:
for line in inf:
if 'Options,' in line:
return False
if line[0] == '#' or line == '\n':
continue
defconfig = line.split()[6] + '_defconfig'
if not os.path.exists(os.path.join(config_dir, defconfig)):
return False
return True
def output_is_new(output, config_dir, srcdir): def output_is_new(output, config_dir, srcdir):
"""Check if the output file is up to date. """Check if the output file is up to date.
@@ -69,13 +115,12 @@ def output_is_new(output, config_dir, srcdir):
srcdir (str): Directory containing Kconfig and MAINTAINERS files srcdir (str): Directory containing Kconfig and MAINTAINERS files
Returns: Returns:
True if the given output file exists and is newer than any of bool: True if the given output file exists and is newer than any of
*_defconfig, MAINTAINERS and Kconfig*. False otherwise. *_defconfig, MAINTAINERS and Kconfig*. False otherwise.
Raises: Raises:
OSError: output file exists but could not be opened OSError: output file exists but could not be opened
""" """
# pylint: disable=too-many-branches
try: try:
ctime = os.path.getctime(output) ctime = os.path.getctime(output)
except OSError as exception: except OSError as exception:
@@ -92,26 +137,10 @@ def output_is_new(output, config_dir, srcdir):
if ctime < os.path.getctime(filepath): if ctime < os.path.getctime(filepath):
return False return False
for (dirpath, _, filenames) in os.walk(srcdir): if not _check_srcdir_is_current(ctime, srcdir):
for filename in filenames:
if (fnmatch.fnmatch(filename, '*~') or
not fnmatch.fnmatch(filename, 'Kconfig*') and
not filename == 'MAINTAINERS'):
continue
filepath = os.path.join(dirpath, filename)
if ctime < os.path.getctime(filepath):
return False return False
# Detect a board that has been removed since the current board database if not _check_output_is_current(output, config_dir):
# was generated
with open(output, encoding="utf-8") as inf:
for line in inf:
if 'Options,' in line:
return False
if line[0] == '#' or line == '\n':
continue
defconfig = line.split()[6] + '_defconfig'
if not os.path.exists(os.path.join(config_dir, defconfig)):
return False return False
return True return True
@@ -134,8 +163,9 @@ class Expr:
Args: Args:
props (list of str): List of properties to check props (list of str): List of properties to check
Returns: Returns:
True if any of the properties match the regular expression bool: True if any of the properties match the regular expression
""" """
for prop in props: for prop in props:
if self._re.match(prop): if self._re.match(prop):
@@ -175,8 +205,9 @@ class Term:
Args: Args:
props (list of str): List of properties to check props (list of str): List of properties to check
Returns: Returns:
True if all of the expressions in the Term match, else False bool: True if all of the expressions in the Term match, else False
""" """
for expr in self._expr_list: for expr in self._expr_list:
if not expr.matches(props): if not expr.matches(props):
@@ -220,6 +251,38 @@ class KconfigScanner:
if self._tmpfile: if self._tmpfile:
try_remove(self._tmpfile) try_remove(self._tmpfile)
def _load_defconfig(self, defconfig):
"""Load a defconfig file, preprocessing if needed
If the defconfig contains #include directives, run the C
preprocessor to expand them before loading.
Args:
defconfig (str): Path to the defconfig file
"""
temp = None
if b'#include' in tools.read_file(defconfig):
cpp = os.getenv('CPP', 'cpp').split()
cmd = cpp + [
'-nostdinc', '-P',
'-I', self._srctree,
'-undef',
'-x', 'assembler-with-cpp',
defconfig]
stdout = command.output(*cmd, capture_stderr=True)
# pylint: disable=consider-using-with
temp = tempfile.NamedTemporaryFile(prefix='buildman-')
tools.write_file(temp.name, stdout, False)
fname = temp.name
tout.info(f'Processing #include to produce {defconfig}')
else:
fname = defconfig
self._conf.load_config(fname)
if temp:
del temp
self._tmpfile = None
def scan(self, defconfig, warn_targets): def scan(self, defconfig, warn_targets):
"""Load a defconfig file to obtain board parameters. """Load a defconfig file to obtain board parameters.
@@ -245,27 +308,7 @@ class KconfigScanner:
expect_target, match, rear = leaf.partition('_defconfig') expect_target, match, rear = leaf.partition('_defconfig')
assert match and not rear, f'{leaf} : invalid defconfig' assert match and not rear, f'{leaf} : invalid defconfig'
temp = None self._load_defconfig(defconfig)
if b'#include' in tools.read_file(defconfig):
cpp = os.getenv('CPP', 'cpp').split()
cmd = cpp + [
'-nostdinc', '-P',
'-I', self._srctree,
'-undef',
'-x', 'assembler-with-cpp',
defconfig]
stdout = command.output(*cmd, capture_stderr=True)
temp = tempfile.NamedTemporaryFile(prefix='buildman-')
tools.write_file(temp.name, stdout, False)
fname = temp.name
tout.info(f'Processing #include to produce {defconfig}')
else:
fname = defconfig
self._conf.load_config(fname)
if temp:
del temp
self._tmpfile = None
params = {} params = {}
warnings = [] warnings = []
@@ -281,22 +324,23 @@ class KconfigScanner:
# Check there is exactly one TARGET_xxx set # Check there is exactly one TARGET_xxx set
if warn_targets: if warn_targets:
target = None warnings += self._check_targets(leaf, expect_target)
for name, sym in self._conf.syms.items():
if name.startswith('TARGET_') and sym.str_value == 'y':
tname = name[7:].lower()
if target:
warnings.append(
f'WARNING: {leaf}: Duplicate TARGET_xxx: {target} and {tname}')
else:
target = tname
if not target:
cfg_name = expect_target.replace('-', '_').upper()
warnings.append(f'WARNING: {leaf}: No TARGET_{cfg_name} enabled')
params['target'] = expect_target params['target'] = expect_target
self._fixup_arch(params)
return params, warnings
def _fixup_arch(self, params):
"""Fix up architecture names
Handle cases where the architecture name needs adjustment based on
CPU type or other configuration.
Args:
params (dict): Board parameters to update in place
"""
# fix-up for aarch64 # fix-up for aarch64
if params['arch'] == 'arm' and params['cpu'] == 'armv8': if params['arch'] == 'arm' and params['cpu'] == 'armv8':
params['arch'] = 'aarch64' params['arch'] = 'aarch64'
@@ -305,14 +349,41 @@ class KconfigScanner:
if params['arch'] == 'riscv': if params['arch'] == 'riscv':
try: try:
value = self._conf.syms.get('ARCH_RV32I').str_value value = self._conf.syms.get('ARCH_RV32I').str_value
except: except AttributeError:
value = '' value = ''
if value == 'y': if value == 'y':
params['arch'] = 'riscv32' params['arch'] = 'riscv32'
else: else:
params['arch'] = 'riscv64' params['arch'] = 'riscv64'
return params, warnings def _check_targets(self, leaf, expect_target):
"""Check that exactly one TARGET_xxx option is set
Args:
leaf (str): Leaf name of defconfig file (for warnings)
expect_target (str): Expected target name
Returns:
list of str: List of warnings found
"""
warnings = []
target = None
for name, sym in self._conf.syms.items():
if name.startswith('TARGET_') and sym.str_value == 'y':
tname = name[7:].lower()
if target:
warnings.append(
f'WARNING: {leaf}: Duplicate TARGET_xxx: '
f'{target} and {tname}')
else:
target = tname
if not target:
cfg_name = expect_target.replace('-', '_').upper()
warnings.append(
f'WARNING: {leaf}: No TARGET_{cfg_name} enabled')
return warnings
class MaintainersDatabase: class MaintainersDatabase:
@@ -380,6 +451,60 @@ class MaintainersDatabase:
self.warnings.append(f"WARNING: no maintainers for '{target}'") self.warnings.append(f"WARNING: no maintainers for '{target}'")
return '' return ''
def _add_targets(self, targets, status, maintainers):
"""Add targets to the database
Args:
targets (list of str): List of target names
status (str): Board status
maintainers (list of str): List of maintainers
"""
for target in targets:
self.database[target] = (status, maintainers)
@staticmethod
def _handle_f_tag(srcdir, rest, targets):
"""Handle F: tag - expand wildcard and filter by defconfig
Args:
srcdir (str): Source directory
rest (str): Remainder of line after 'F:'
targets (list of str): List to append targets to
"""
glob_path = os.path.join(srcdir, rest)
for item in glob.glob(glob_path):
front, match, rear = item.partition('configs/')
if front.endswith('/'):
front = front[:-1]
if front == srcdir and match:
front, match, rear = rear.rpartition('_defconfig')
if match and not rear:
targets.append(front)
@staticmethod
def _handle_n_tag(srcdir, rest, targets):
"""Handle N: tag - scan configs dir and match with regex
Args:
srcdir (str): Source directory
rest (str): Remainder of line after 'N:'
targets (list of str): List to append targets to
"""
walk_path = os.walk(os.path.join(srcdir, 'configs'))
for dirpath, _, fnames in walk_path:
for cfg in fnames:
path = os.path.join(dirpath, cfg)[len(srcdir) + 1:]
front, match, rear = path.partition('configs/')
if front or not match:
continue
front, match, rear = rear.rpartition('_defconfig')
# Use this entry if it matches the defconfig file
# without the _defconfig suffix. For example
# 'am335x.*' matches am335x_guardian_defconfig
if match and not rear and re.search(rest, front):
targets.append(front)
def parse_file(self, srcdir, fname): def parse_file(self, srcdir, fname):
"""Parse a MAINTAINERS file. """Parse a MAINTAINERS file.
@@ -397,21 +522,11 @@ class MaintainersDatabase:
srcdir (str): Directory containing source code (Kconfig files) srcdir (str): Directory containing source code (Kconfig files)
fname (str): MAINTAINERS file to be parsed fname (str): MAINTAINERS file to be parsed
""" """
def add_targets(linenum):
"""Add any new targets
Args:
linenum (int): Current line number
"""
if targets:
for target in targets:
self.database[target] = (status, maintainers)
targets = [] targets = []
maintainers = [] maintainers = []
status = '-' status = '-'
with open(fname, encoding="utf-8") as inf: with open(fname, encoding="utf-8") as inf:
for linenum, line in enumerate(inf): for _, line in enumerate(inf):
# Check also commented maintainers # Check also commented maintainers
if line[:3] == '#M:': if line[:3] == '#M:':
line = line[1:] line = line[1:]
@@ -419,41 +534,17 @@ class MaintainersDatabase:
if tag == 'M:': if tag == 'M:':
maintainers.append(rest) maintainers.append(rest)
elif tag == 'F:': elif tag == 'F:':
# expand wildcard and filter by 'configs/*_defconfig' self._handle_f_tag(srcdir, rest, targets)
glob_path = os.path.join(srcdir, rest)
for item in glob.glob(glob_path):
front, match, rear = item.partition('configs/')
if front.endswith('/'):
front = front[:-1]
if front == srcdir and match:
front, match, rear = rear.rpartition('_defconfig')
if match and not rear:
targets.append(front)
elif tag == 'S:': elif tag == 'S:':
status = rest status = rest
elif tag == 'N:': elif tag == 'N:':
# Just scan the configs directory since that's all we care self._handle_n_tag(srcdir, rest, targets)
# about
walk_path = os.walk(os.path.join(srcdir, 'configs'))
for dirpath, _, fnames in walk_path:
for cfg in fnames:
path = os.path.join(dirpath, cfg)[len(srcdir) + 1:]
front, match, rear = path.partition('configs/')
if front or not match:
continue
front, match, rear = rear.rpartition('_defconfig')
# Use this entry if it matches the defconfig file
# without the _defconfig suffix. For example
# 'am335x.*' matches am335x_guardian_defconfig
if match and not rear and re.search(rest, front):
targets.append(front)
elif line == '\n': elif line == '\n':
add_targets(linenum) self._add_targets(targets, status, maintainers)
targets = [] targets = []
maintainers = [] maintainers = []
status = '-' status = '-'
add_targets(linenum) self._add_targets(targets, status, maintainers)
class Boards: class Boards:
@@ -502,7 +593,7 @@ class Boards:
"""Return a list of available boards. """Return a list of available boards.
Returns: Returns:
List of Board objects list of Board: List of Board objects
""" """
return self._boards return self._boards
@@ -523,7 +614,8 @@ class Boards:
"""Return a dictionary containing the selected boards """Return a dictionary containing the selected boards
Returns: Returns:
List of Board objects that are marked selected OrderedDict: Boards that are marked selected (key=target,
value=Board)
""" """
board_dict = OrderedDict() board_dict = OrderedDict()
for brd in self._boards: for brd in self._boards:
@@ -535,7 +627,7 @@ class Boards:
"""Return a list of selected boards """Return a list of selected boards
Returns: Returns:
List of Board objects that are marked selected list of Board: Board objects that are marked selected
""" """
return [brd for brd in self._boards if brd.build_it] return [brd for brd in self._boards if brd.build_it]
@@ -543,7 +635,7 @@ class Boards:
"""Return a list of selected boards """Return a list of selected boards
Returns: Returns:
List of board names that are marked selected list of str: Board names that are marked selected
""" """
return [brd.target for brd in self._boards if brd.build_it] return [brd.target for brd in self._boards if brd.build_it]
@@ -598,34 +690,12 @@ class Boards:
terms.append(term) terms.append(term)
return terms return terms
def select_boards(self, args, exclude=None, brds=None): @staticmethod
"""Mark boards selected based on args # pylint: disable=too-many-arguments
def _check_board(brd, terms, brds, found, exclude_list, result):
Normally either boards (an explicit list of boards) or args (a list of
terms to match against) is used. It is possible to specify both, in
which case they are additive.
If brds and args are both empty, all boards are selected.
Args:
args (list of str): List of strings specifying boards to include,
either named, or by their target, architecture, cpu, vendor or
soc. If empty, all boards are selected.
exclude (list of str): List of boards to exclude, regardless of
'args', or None for none
brds (list of Board): List of boards to build, or None/[] for all
Returns:
Tuple
Dictionary which holds the list of boards which were selected
due to each argument, arranged by argument.
List of errors found
"""
def _check_board(brd):
"""Check whether to include or exclude a board """Check whether to include or exclude a board
Checks the various terms and decide whether to build it or not (the Checks the various terms and decides whether to build it or not.
'build_it' variable).
If it is built, add the board to the result[term] list so we know If it is built, add the board to the result[term] list so we know
which term caused it to be built. Add it to result['all'] also. which term caused it to be built. Add it to result['all'] also.
@@ -635,6 +705,11 @@ class Boards:
Args: Args:
brd (Board): Board to check brd (Board): Board to check
terms (list of Term): Terms to match against
brds (list of str): List of board names to build, or None
found (list of str): List to append found board names to
exclude_list (list of Expr): Expressions for boards to exclude
result (OrderedDict): Dict to store results in
""" """
matching_term = None matching_term = None
build_it = False build_it = False
@@ -663,6 +738,29 @@ class Boards:
result[matching_term].append(brd.target) result[matching_term].append(brd.target)
result['all'].append(brd.target) result['all'].append(brd.target)
def select_boards(self, args, exclude=None, brds=None):
"""Mark boards selected based on args
Normally either boards (an explicit list of boards) or args (a list of
terms to match against) is used. It is possible to specify both, in
which case they are additive.
If brds and args are both empty, all boards are selected.
Args:
args (list of str): List of strings specifying boards to include,
either named, or by their target, architecture, cpu, vendor or
soc. If empty, all boards are selected.
exclude (list of str): List of boards to exclude, regardless of
'args', or None for none
brds (list of Board): List of boards to build, or None/[] for all
Returns:
tuple:
OrderedDict: Boards selected due to each argument, keyed by
argument
list of str: Errors/warnings found
"""
result = OrderedDict() result = OrderedDict()
warnings = [] warnings = []
terms = self._build_terms(args) terms = self._build_terms(args)
@@ -678,7 +776,7 @@ class Boards:
found = [] found = []
for brd in self._boards: for brd in self._boards:
_check_board(brd) self._check_board(brd, terms, brds, found, exclude_list, result)
if brds: if brds:
remaining = set(brds) - set(found) remaining = set(brds) - set(found)
@@ -723,6 +821,55 @@ class Boards:
params_list.append(params) params_list.append(params)
warnings.update(warn) warnings.update(warn)
@staticmethod
def _collect_defconfigs(config_dir):
"""Collect all defconfig files from a directory
Args:
config_dir (str): Directory containing the defconfig files
Returns:
list of str: Paths to all defconfig files found
"""
all_defconfigs = []
for (dirpath, _, filenames) in os.walk(config_dir):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
all_defconfigs.append(os.path.join(dirpath, filename))
return all_defconfigs
def _start_scanners(self, all_defconfigs, srcdir, jobs, warn_targets):
"""Start parallel defconfig scanning processes
Args:
all_defconfigs (list of str): Paths to defconfig files to scan
srcdir (str): Directory containing source code (Kconfig files)
jobs (int): The number of jobs to run simultaneously
warn_targets (bool): True to warn about missing or duplicate
CONFIG_TARGET options
Returns:
tuple:
list of Process: Running scanner processes
list of Queue: Queues for receiving results
"""
total_boards = len(all_defconfigs)
processes = []
queues = []
for i in range(jobs):
defconfigs = all_defconfigs[total_boards * i // jobs :
total_boards * (i + 1) // jobs]
que = multiprocessing.Queue(maxsize=-1)
proc = multiprocessing.Process(
target=self.scan_defconfigs_for_multiprocess,
args=(srcdir, que, defconfigs, warn_targets))
proc.start()
processes.append(proc)
queues.append(que)
return processes, queues
def scan_defconfigs(self, config_dir, srcdir, jobs=1, warn_targets=False): def scan_defconfigs(self, config_dir, srcdir, jobs=1, warn_targets=False):
"""Collect board parameters for all defconfig files. """Collect board parameters for all defconfig files.
@@ -743,33 +890,16 @@ class Boards:
value: string value of the key value: string value of the key
list of str: List of warnings recorded list of str: List of warnings recorded
""" """
all_defconfigs = [] all_defconfigs = self._collect_defconfigs(config_dir)
for (dirpath, _, filenames) in os.walk(config_dir): processes, queues = self._start_scanners(all_defconfigs, srcdir, jobs,
for filename in fnmatch.filter(filenames, '*_defconfig'): warn_targets)
if fnmatch.fnmatch(filename, '.*'):
continue
all_defconfigs.append(os.path.join(dirpath, filename))
total_boards = len(all_defconfigs)
processes = []
queues = []
for i in range(jobs):
defconfigs = all_defconfigs[total_boards * i // jobs :
total_boards * (i + 1) // jobs]
que = multiprocessing.Queue(maxsize=-1)
proc = multiprocessing.Process(
target=self.scan_defconfigs_for_multiprocess,
args=(srcdir, que, defconfigs, warn_targets))
proc.start()
processes.append(proc)
queues.append(que)
# The resulting data should be accumulated to these lists # The resulting data should be accumulated to these lists
params_list = [] params_list = []
warnings = set() warnings = set()
# Data in the queues should be retrieved preriodically. # Data in the queues should be retrieved preriodically. Otherwise,
# Otherwise, the queues would become full and subprocesses would get stuck. # the queues would become full and subprocesses would get stuck.
while any(p.is_alive() for p in processes): while any(p.is_alive() for p in processes):
self.read_queues(queues, params_list, warnings) self.read_queues(queues, params_list, warnings)
# sleep for a while until the queues are filled # sleep for a while until the queues are filled
@@ -790,6 +920,7 @@ class Boards:
"""Add Status and Maintainers information to the board parameters list. """Add Status and Maintainers information to the board parameters list.
Args: Args:
srcdir (str): Directory containing source code (MAINTAINERS files)
params_list (list of dict): A list of the board parameters params_list (list of dict): A list of the board parameters
Returns: Returns:
@@ -884,7 +1015,8 @@ class Boards:
output (str): The name of the output file output (str): The name of the output file
jobs (int): The number of jobs to run simultaneously jobs (int): The number of jobs to run simultaneously
force (bool): Force to generate the output even if it is new force (bool): Force to generate the output even if it is new
quiet (bool): True to avoid printing a message if nothing needs doing quiet (bool): True to avoid printing a message if nothing needs
doing
Returns: Returns:
bool: True if all is well, False if there were warnings bool: True if all is well, False if there were warnings
@@ -948,7 +1080,12 @@ class Boards:
raise ValueError(f"Board '{target}' not found") raise ValueError(f"Board '{target}' not found")
def parse_extended(self, dbase, fname): def parse_extended(self, dbase, fname):
"""Parse a single 'extended' file""" """Parse a single 'extended' file
Args:
dbase (tuple): Database of defconfigs from qconfig
fname (str): Path to the extended-board file to parse
"""
result = ExtendedParser.parse_file(fname) result = ExtendedParser.parse_file(fname)
for ext in result: for ext in result:
ext_boards = self.scan_extended(dbase, ext) ext_boards = self.scan_extended(dbase, ext)
@@ -963,7 +1100,15 @@ class Boards:
self.add_board(newb) self.add_board(newb)
def scan_extended(self, dbase, ext): def scan_extended(self, dbase, ext):
"""Scan for extended boards""" """Scan for extended boards
Args:
dbase (tuple): Database of defconfigs
ext (Extended): Extended-board definition
Returns:
set of str: Set of board names matching the extended definition
"""
# First check the fragments # First check the fragments
frags = [] frags = []
for frag in ext.fragments: for frag in ext.fragments:
@@ -1025,13 +1170,28 @@ class ExtendedParser:
@staticmethod @staticmethod
def parse_file(fname): def parse_file(fname):
"""Parse a file and return the result""" """Parse a file and return the result
Args:
fname (str): Filename to parse
Returns:
list of Extended: List of extended-board definitions
"""
return ExtendedParser.parse_data(fname, return ExtendedParser.parse_data(fname,
tools.read_file(fname, binary=False)) tools.read_file(fname, binary=False))
@staticmethod @staticmethod
def parse_data(fname, data): def parse_data(fname, data):
"""Parse a file and return the result""" """Parse a file and return the result
Args:
fname (str): Filename (for error messages)
data (str): Contents of the file
Returns:
list of Extended: List of extended-board definitions
"""
parser = ExtendedParser() parser = ExtendedParser()
parser.parse(fname, data) parser.parse(fname, data)
return parser.extended return parser.extended
@@ -1042,6 +1202,12 @@ class ExtendedParser:
Args: Args:
fname (str): Filename to parse (used for error messages) fname (str): Filename to parse (used for error messages)
data (str): Contents of the file data (str): Contents of the file
Returns:
list of Extended: List of extended-board definitions
Raises:
ValueError: Invalid syntax in file
""" """
self.start() self.start()
for seq, line in enumerate(data.splitlines()): for seq, line in enumerate(data.splitlines()):
@@ -1054,7 +1220,8 @@ class ExtendedParser:
if '=' in line: if '=' in line:
pair = line.split('=') pair = line.split('=')
if len(pair) != 2: if len(pair) != 2:
raise ValueError(f'{fname}:{linenum}: Invalid CONFIG syntax') raise ValueError(
f'{fname}:{linenum}: Invalid CONFIG syntax')
first, rest = pair first, rest = pair
cfg = first.strip() cfg = first.strip()
value = rest.strip() value = rest.strip()
@@ -1062,9 +1229,26 @@ class ExtendedParser:
else: else:
target = line.strip() target = line.strip()
if ' ' in target: if ' ' in target:
raise ValueError(f'{fname}:{linenum}: Invalid target regex') raise ValueError(
f'{fname}:{linenum}: Invalid target regex')
self.targets.append(['regex', line.strip()]) self.targets.append(['regex', line.strip()])
else: else:
self._parse_tag(fname, linenum, line)
self.finish()
return self.extended
def _parse_tag(self, fname, linenum, line):
"""Parse a tag line (one not starting with a space)
Args:
fname (str): Filename (for error messages)
linenum (int): Line number (for error messages)
line (str): Line to parse
Raises:
ValueError: Invalid syntax
"""
pair = line.split(':') pair = line.split(':')
if len(pair) != 2: if len(pair) != 2:
raise ValueError(f'{fname}:{linenum}: Invalid tag') raise ValueError(f'{fname}:{linenum}: Invalid tag')
@@ -1083,6 +1267,3 @@ class ExtendedParser:
self.in_targets = True self.in_targets = True
else: else:
raise ValueError(f"{fname}:{linenum}: Unknown tag '{tag}'") raise ValueError(f"{fname}:{linenum}: Unknown tag '{tag}'")
self.finish()
return self.extended