From 3a956c5f69873611ae5f8dcb9acd117f66b95ddc Mon Sep 17 00:00:00 2001 From: Eli Schwartz Date: Tue, 25 Mar 2025 16:08:48 -0400 Subject: [PATCH 1/8] meson: disable coccinelle configuration when building from a tarball Wiring up coccinelle in the build, depends on running git commands to get the list of files to operate on. Reasonable, for a feature mainly used by people developing on git. If building git itself from a tarball distribution of git's own source code, one likely does not need to run coccinelle. But running those git commands failed, and caused the build to error out, if `spatch` was installed -- because the build assumed that its presence indicated a desire to use it on this source tree. Instead, we can expand the conditional to check for both `spatch` and the `.git` file or directory. Meson's `opt.require()` method allows us to add a prerequisite for the feature option. If the prerequisite fails, then the option either: - converts autodetection to disabled - emits an informative error if the feature was set to enabled: ``` ERROR: Feature coccinelle cannot be enabled: coccinelle can only be run from a git checkout ``` Signed-off-by: Eli Schwartz Signed-off-by: Junio C Hamano --- contrib/coccinelle/meson.build | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/contrib/coccinelle/meson.build b/contrib/coccinelle/meson.build index 5d76a7fee6fbb6..ea054c924f400f 100644 --- a/contrib/coccinelle/meson.build +++ b/contrib/coccinelle/meson.build @@ -1,4 +1,9 @@ -spatch = find_program('spatch', required: get_option('coccinelle')) +coccinelle_opt = get_option('coccinelle').require( + fs.exists(meson.project_source_root() / '.git'), + error_message: 'coccinelle can only be run from a git checkout', +) + +spatch = find_program('spatch', required: coccinelle_opt) if not spatch.found() subdir_done() endif From 53ce389a9ead9329dc0392dfcf7d58a1cd9522f3 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Sun, 30 Mar 2025 22:49:03 +0200 Subject: [PATCH 2/8] meson: add corresponding target for Makefile's hdr-check To bridge the remaining gaps between Makefile and Meson, this patch series adds 'hdr-check' to Meson to compliment the Makefile's 'hdr-check'. We also introduce 'headers-check' as an alias to 'hdr-check' as a better named replacement in both Meson and make and add a note to deprecate 'hdr-check' in the future. The first two commits are small cleanups, where we re-organize existing variables to make it easier to add the target. The third commit adds the 'hdr-check' target to Meson. The last commit introduces the 'headers-check' alias to both Meson and the makefile and marks 'hdr-check' to be deprecated. This is based on master 9d22ac5122 (The third batch, 2025-04-07) with 'es/meson-build-skip-coccinelle' merged in. --- Changes in v5: - Add a commit to install 'git' in GitHub's CI before the repository is checked out. Without the presence of the 'git' executable, GitHub downloads a tar of the repo instead of cloning it. This causes the patch series to fail in CI. - Expose the 'headers_to_check' variable even if 'git' executable is not found or not a repository. This ensures dependencies of 'headers_to_check' can simply rely on its length. We also check that the '.git' folder is setup before populating 'headers_to_check', this ensures Meson doesn't fail in tarball of the Git source code. - Link to v4: https://lore.kernel.org/r/20250420-505-wire-up-sparse-via-meson-v4-0-66e14134e822@gmail.com Changes in v4: - Rename headers to headers_to_check, since these headers are only used for static analysis. - Added a commit to rename third_party_sources -> third_party_excludes and remove a duplicate. - Fix a typo 'gcrpyt' -> 'gcrypt' - Remove 'generated_headers', since we use 'git ls-files' and that would already ignore files within '.gitignore'. - Link to v3: https://lore.kernel.org/r/20250414-505-wire-up-sparse-via-meson-v3-0-edc6e7f26745@gmail.com Changes in v3: - Some renames: - headers_generated -> generated_headers - meson -> Meson - headers-check -> check-headers - headers_check_exclude -> exclude_from_check_headers - Rewrite 'headers_check_exclude' to also contain dirs so we can skip listing individual header files. - Move 'xdiff/*' to 'third_party_sources' and cleanup 'exclude_from_check_headers'. - Use 'echo' instead of 'echo -n'. - Use `fs.replace_suffix` instead of `str.replace`. - Link to v2: https://lore.kernel.org/r/20250410-505-wire-up-sparse-via-meson-v2-0-acb45cc8a2e5@gmail.com Changes in v2: - Add 'hdr-check' to meson, while introducing 'headers-check' as a replacement alias. Schedule 'hdr-check' to be deprecated in the future. - Link to v1: https://lore.kernel.org/r/20250408-505-wire-up-sparse-via-meson-v1-0-17476e5cea3f@gmail.com --- b4-submit-tracking --- { "series": { "revision": 5, "change-id": "20250330-505-wire-up-sparse-via-meson-2e32dd31208b", "prefixes": [], "history": { "v1": [ "20250408-505-wire-up-sparse-via-meson-v1-0-17476e5cea3f@gmail.com" ], "v2": [ "20250410-505-wire-up-sparse-via-meson-v2-0-acb45cc8a2e5@gmail.com" ], "v3": [ "20250414-505-wire-up-sparse-via-meson-v3-0-edc6e7f26745@gmail.com" ], "v4": [ "20250420-505-wire-up-sparse-via-meson-v4-0-66e14134e822@gmail.com" ] } } } From d20993a59ec8545404be31280eb08c20d752e467 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Mon, 21 Apr 2025 13:06:45 +0200 Subject: [PATCH 3/8] ci/github: install git before checking out the repository The GitHub's CI workflow uses 'actions/checkout@v4' to checkout the repository. This action defaults to using the GitHub REST API to obtain the repository if the `git` executable isn't available. The step to build Git in the GitHub workflow can be summarized as: ... - uses: actions/checkout@v4 #1 - run: ci/install-dependencies.sh #2 ... - run: sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh #3 ... Step #1, clones the repository, since the `git` executable isn't present at this step, it uses GitHub's REST API to obtain a tar of the repository. Step #2, installs all dependencies, which includes the `git` executable. Step #3, sets up the build, which includes setting up meson in the meson job. At this point the `git` executable is present. This means while the `git` executable is present, the repository doesn't contain the '.git' folder. To keep both the CI's (GitLab and GitHub) behavior consistent and to ensure that the build is performed on a real-world scenario, install `git` before the repository is checked out. This ensures that 'actions/checkout@v4' will clone the repository instead of using a tarball. We also update the package cache while installing `git`, this is because some distros will fail to locate the package without updating the cache. Helped-by: Phillip Wood Signed-off-by: Karthik Nayak --- .github/workflows/main.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 37541f3d10daba..e9112b3a64cd52 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -414,6 +414,20 @@ jobs: - name: prepare libc6 for actions if: matrix.vector.jobname == 'linux32' run: apt -q update && apt -q -y install libc6-amd64 lib64stdc++6 + - name: install git in container + run: | + if command -v git + then + : # nothing to do + elif command -v apk + then + apk add --update git + elif command -v dnf + then + dnf -yq update && dnf -yq install git + else + apt-get -q update && apt-get -q -y install git + fi - uses: actions/checkout@v4 - run: ci/install-dependencies.sh - run: useradd builder --create-home From 6eb10dd3b105b305b5ed5ea7e6df01623762d95c Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Mon, 7 Apr 2025 11:26:18 +0200 Subject: [PATCH 4/8] coccinelle: meson: rename variables to be more specific In Meson, included subdirs export their variables to top level Meson builds. In 'contrib/coccinelle/meson.build', we define two such variables `sources` and `headers`. While these variables are specific to the checks in the 'contrib/coccinelle/' directory, they also pollute the top level 'meson.build'. Rename them to be more specific, this ensures that they aren't mistakenly used in the upper levels and avoid variable name collisions. While here, change the empty list denotation to be consistent with other places. Signed-off-by: Karthik Nayak --- contrib/coccinelle/meson.build | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contrib/coccinelle/meson.build b/contrib/coccinelle/meson.build index ea054c924f400f..03ce52d7522b76 100644 --- a/contrib/coccinelle/meson.build +++ b/contrib/coccinelle/meson.build @@ -55,18 +55,18 @@ concatenated_rules = custom_target( capture: true, ) -sources = [ ] +coccinelle_sources = [] foreach source : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.c', third_party_sources, check: true).stdout().split() - sources += source + coccinelle_sources += source endforeach -headers = [ ] +coccinelle_headers = [] foreach header : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.h', third_party_sources, check: true).stdout().split() - headers += meson.project_source_root() / header + coccinelle_headers += meson.project_source_root() / header endforeach patches = [ ] -foreach source : sources +foreach source : coccinelle_sources patches += custom_target( command: [ spatch, @@ -78,7 +78,7 @@ foreach source : sources input: meson.project_source_root() / source, output: source.underscorify() + '.patch', capture: true, - depend_files: headers, + depend_files: coccinelle_headers, ) endforeach From 7950b44c6f7ea2471e56e258201a71a9cfca3175 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Mon, 7 Apr 2025 11:29:02 +0200 Subject: [PATCH 5/8] meson: move headers definition from 'contrib/coccinelle' The Meson build for coccinelle static analysis lists all headers to analyse. Due to the way Meson exports variables between subdirs, this variable is also available in the root Meson build. An upcoming commit, will add a new check complimenting 'hdr-check' in the Makefile. This would require the list of headers. So move the 'coccinelle_headers' to the root Meson build and rename it to 'headers', remove the root path being appended to each header and retain that in the coccinelle Meson build since it is specific to the coccinelle build. Also move the 'third_party_sources' variable to the root Meson build since it is also a dependency for the 'headers' variable. This also makes it easier to understand as the variable is now propagated from the top level to the bottom. While 'headers_to_check' is only computed when we have a repository and the 'git' executable is present, the variable itself is exposed as an empty array. This allows dependencies in upcoming commits to simply check for length of the array and not worry about dependencies required to actually populate the array. Signed-off-by: Karthik Nayak --- contrib/coccinelle/meson.build | 17 +---------------- meson.build | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+), 16 deletions(-) diff --git a/contrib/coccinelle/meson.build b/contrib/coccinelle/meson.build index 03ce52d7522b76..4f07824402f317 100644 --- a/contrib/coccinelle/meson.build +++ b/contrib/coccinelle/meson.build @@ -8,21 +8,6 @@ if not spatch.found() subdir_done() endif -third_party_sources = [ - ':!contrib', - ':!compat/inet_ntop.c', - ':!compat/inet_pton.c', - ':!compat/nedmalloc', - ':!compat/obstack.*', - ':!compat/poll', - ':!compat/regex', - ':!sha1collisiondetection', - ':!sha1dc', - ':!t/unit-tests/clar', - ':!t/unit-tests/clar', - ':!t/t[0-9][0-9][0-9][0-9]*', -] - rules = [ 'array.cocci', 'commit.cocci', @@ -61,7 +46,7 @@ foreach source : run_command(git, '-C', meson.project_source_root(), 'ls-files', endforeach coccinelle_headers = [] -foreach header : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.h', third_party_sources, check: true).stdout().split() +foreach header : headers_to_check coccinelle_headers += meson.project_source_root() / header endforeach diff --git a/meson.build b/meson.build index e98cfa4909f288..e147ddff286bec 100644 --- a/meson.build +++ b/meson.build @@ -633,6 +633,28 @@ builtin_sources = [ 'builtin/write-tree.c', ] +third_party_sources = [ + ':!contrib', + ':!compat/inet_ntop.c', + ':!compat/inet_pton.c', + ':!compat/nedmalloc', + ':!compat/obstack.*', + ':!compat/poll', + ':!compat/regex', + ':!sha1collisiondetection', + ':!sha1dc', + ':!t/unit-tests/clar', + ':!t/unit-tests/clar', + ':!t/t[0-9][0-9][0-9][0-9]*', +] + +headers_to_check = [] +if git.found() and fs.exists(meson.project_source_root() / '.git') + foreach header : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.h', third_party_sources, check: true).stdout().split() + headers_to_check += header + endforeach +endif + if not get_option('breaking_changes') builtin_sources += 'builtin/pack-redundant.c' endif From 7fd64b788b2a9f7f02c79ad7408b29d598348350 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Sun, 20 Apr 2025 12:05:25 +0200 Subject: [PATCH 6/8] meson: rename 'third_party_sources' to 'third_party_excludes' The 'third_party_sources' variable was moved to the root 'meson.build' file in the previous commit. The variable is actually used to exclude third party sources, so rename it accordingly to 'third_party_excludes' to avoid confusion. While here, remove a duplicate from the list. Signed-off-by: Karthik Nayak --- contrib/coccinelle/meson.build | 2 +- meson.build | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/contrib/coccinelle/meson.build b/contrib/coccinelle/meson.build index 4f07824402f317..dc3f73c2e7b117 100644 --- a/contrib/coccinelle/meson.build +++ b/contrib/coccinelle/meson.build @@ -41,7 +41,7 @@ concatenated_rules = custom_target( ) coccinelle_sources = [] -foreach source : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.c', third_party_sources, check: true).stdout().split() +foreach source : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.c', third_party_excludes, check: true).stdout().split() coccinelle_sources += source endforeach diff --git a/meson.build b/meson.build index e147ddff286bec..4618804c7a19b1 100644 --- a/meson.build +++ b/meson.build @@ -633,7 +633,7 @@ builtin_sources = [ 'builtin/write-tree.c', ] -third_party_sources = [ +third_party_excludes = [ ':!contrib', ':!compat/inet_ntop.c', ':!compat/inet_pton.c', @@ -644,13 +644,12 @@ third_party_sources = [ ':!sha1collisiondetection', ':!sha1dc', ':!t/unit-tests/clar', - ':!t/unit-tests/clar', ':!t/t[0-9][0-9][0-9][0-9]*', ] headers_to_check = [] if git.found() and fs.exists(meson.project_source_root() / '.git') - foreach header : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.h', third_party_sources, check: true).stdout().split() + foreach header : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.h', third_party_excludes, check: true).stdout().split() headers_to_check += header endforeach endif From df0472be57c36699c2aad9a2e7a84c6b16c00103 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Mon, 7 Apr 2025 12:46:37 +0200 Subject: [PATCH 7/8] meson: add support for 'hdr-check' The Makefile supports a target called 'hdr-check', which checks if individual header files can be independently compiled. Let's port this functionality to Meson, our new build system too. The implementation resembles that of the Makefile and provides the same check. Since meson builds are out-of-tree, header dependencies are not automatically met. So unlike the Makefile version, we also need to add the required dependencies. Also add the 'xdiff/' dir to the list of 'third_party_sources' as those headers must be skipped from the checks too. This also skips the folder from the 'coccinelle' checks, this is okay, since this code is an external dependency. Signed-off-by: Karthik Nayak --- meson.build | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/meson.build b/meson.build index 4618804c7a19b1..22fc65ec80ba6a 100644 --- a/meson.build +++ b/meson.build @@ -645,6 +645,7 @@ third_party_excludes = [ ':!sha1dc', ':!t/unit-tests/clar', ':!t/t[0-9][0-9][0-9][0-9]*', + ':!xdiff', ] headers_to_check = [] @@ -1994,6 +1995,68 @@ endif subdir('contrib') +exclude_from_check_headers = [ + 'compat/', + 'unicode-width.h', +] + +if sha1_backend != 'openssl' + exclude_from_check_headers += 'sha1/openssl.h' +endif +if sha256_backend != 'openssl' + exclude_from_check_headers += 'sha256/openssl.h' +endif +if sha256_backend != 'nettle' + exclude_from_check_headers += 'sha256/nettle.h' +endif +if sha256_backend != 'gcrypt' + exclude_from_check_headers += 'sha256/gcrypt.h' +endif + +if headers_to_check.length() != 0 and compiler.get_argument_syntax() == 'gcc' + hco_targets = [] + foreach h : headers_to_check + skip_header = false + foreach exclude : exclude_from_check_headers + if h.startswith(exclude) + skip_header = true + break + endif + endforeach + + if skip_header + continue + endif + + hcc = custom_target( + input: h, + output: h.underscorify() + 'cc', + command: [ + shell, + '-c', + 'echo \'#include "git-compat-util.h"\' > @OUTPUT@ && echo \'#include "' + h + '"\' >> @OUTPUT@' + ] + ) + + hco = custom_target( + input: hcc, + output: fs.replace_suffix(h.underscorify(), '.hco'), + command: [ + compiler.cmd_array(), + libgit_c_args, + '-I', meson.project_source_root(), + '-I', meson.project_source_root() / 't/unit-tests', + '-o', '/dev/null', + '-c', '-xc', + '@INPUT@' + ] + ) + hco_targets += hco + endforeach + + alias_target('hdr-check', hco_targets) +endif + foreach key, value : { 'DIFF': diff.full_path(), 'GIT_SOURCE_DIR': meson.project_source_root(), From 7d9a33767be4514a2ff29a9e54f44a224fa92bb1 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Thu, 10 Apr 2025 12:45:04 +0200 Subject: [PATCH 8/8] makefile/meson: add 'check-headers' as alias for 'hdr-check' The 'hdr-check' target in Meson and makefile is used to check if headers can be compiled individually. The naming however isn't readable as 'hdr' is not a common shortforme for 'header', neither is it an abbreviation. Let's introduce 'check-headers' as an alternative target for 'hdr-check' and add a `TODO` to deprecate the latter after 2 releases. Since this is an internal tool, we can use a shorter deprecation cycle. Change existing usage of 'hdr-check' in 'ci/run-static-analysis.sh' to also use 'check-headers'. Signed-off-by: Karthik Nayak --- Makefile | 4 +++- ci/run-static-analysis.sh | 2 +- meson.build | 4 +++- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index ac32d2d0bdae93..961ee508be37c4 100644 --- a/Makefile +++ b/Makefile @@ -3326,8 +3326,10 @@ HCC = $(HCO:hco=hcc) $(HCO): %.hco: %.hcc $(GENERATED_H) FORCE $(QUIET_HDR)$(CC) $(ALL_CFLAGS) -o /dev/null -c -xc $< -.PHONY: hdr-check $(HCO) +# TODO: deprecate 'hdr-check' in lieu of 'check-headers' in Git 2.51+ +.PHONY: hdr-check check-headers $(HCO) hdr-check: $(HCO) +check-headers: hdr-check .PHONY: style style: diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh index 0d51e5ce0e7cb4..60c175a094331d 100755 --- a/ci/run-static-analysis.sh +++ b/ci/run-static-analysis.sh @@ -26,7 +26,7 @@ then exit 1 fi -make hdr-check || +make check-headers || exit 1 make check-pot diff --git a/meson.build b/meson.build index 22fc65ec80ba6a..569e3888fb2d0b 100644 --- a/meson.build +++ b/meson.build @@ -2054,7 +2054,9 @@ if headers_to_check.length() != 0 and compiler.get_argument_syntax() == 'gcc' hco_targets += hco endforeach - alias_target('hdr-check', hco_targets) + # TODO: deprecate 'hdr-check' in lieu of 'check-headers' in Git 2.51+ + hdr_check = alias_target('hdr-check', hco_targets) + alias_target('check-headers', hdr_check) endif foreach key, value : {