Perl core includes a "HTTP::Tiny" module that does what little we need,
and has had it since 5.14 or so, so prefer that instead and remove the
requirement to install libwww-perl.
This also required removing the one minor instance of the URI module,
which is not technically in Perl core but was installed due to LWP.
Instead we just use a regexp 'blessed' by the URI module author.
There are currently various different paths to creating a Module (the
internal representation of a module from the rc-file). Depending on how
the Module is created you might get different options set for the Module
since the options are only available near the beginning (and are spread
across many different variables).
Consolidate some of this; this also has the side benefit of making
ksb::Application less of a dumping ground, and let's us get rid of a
'module-option-fixing callback' that needed to be used with a separate
'module-creating-by-name' callback.
Additionally this means that modules take only the options of the
module-set they were actually declared in, instead of the module-set in
use when the module was found (which may be different when using
--include-dependencies).
BUG:345192
FIXED-IN:15.05
I noticed that I was getting error messages from a git usage after
install on my KF5 modules, which are configured to use
remove-after-install.
Turns out that git or the shell or kernel or something doesn't like
running a command from a now-invalid cwd.
The recent(-ish) switch to allow short name from a few months ago in the
dependency-data files was buggy in the way it handles "catch all"
dependencies.
Dependency information was already shortened down to just the module
name when it was compared against the wildcard dependency rules, which
made it impossible for most of the rules to match (e.g. a dependency for
kde/kdemultimedia/* wouldn't match against 'juk' anymore because at that
portion of the code it's not clear that juk is actually a part of that
set).
I will still have to make some other fixes to permit wildcarded
dependencies to work through indirect modules. E.g. a dependency exists
for kde/foo/bar/bar* to "first-module". A second module depends on
"bar-intermedia-module", which is really under kde/foo/bar/. If
"bar-intermedia-module" is actually being built this is not an issue as
the dependency resolver can read the full path to detect the recursive
dependency onto "first-module". But if "bar-intermedia-module" is not
being built then there's no way to tell that there's really a dependency
on "first-module" since the wildcard dependency would not match.
Instead of requiring wildcards to immediately follow a '/' character in
branch-group handling (or to be a bare '*'), allow for wildcards
following other characters since it seems CI supports that syntax. This
should fix branch selection for the KDE Telepathy modules.
My last commit checked for missing executables before trying to execute
them, but still left open the possibility that it would be impossible to
call exec(2) on an executable that had just been found (for whatever
reason).
Closer inspection in 'perldoc perlipc', and 'perldoc -f open' has helped
illustrate how to catch that condition. E.g. it turns out that calling
'close' on a pipe-opened filehandle will automatically call waitpid,
plus I'd forgotten to exit in that error case in the child proc. (This
is why it would take 2-3 seconds after failing for the parent process to
continue).
The more interesting question is what error policy to use in this case.
Errors are expected in pretend mode (this function actually runs
processes even during pretend mode since otherwise it can be impossible
to determine whether to clone or update, switch branches, etc.), but on
the other hand during execution errors should be unexpected and
noteworthy.
FIXED-IN:15.04
BUG:345425
Perl's "open '-|' $program" feature, which is used to pipe the output of
a process to a filehandle variable, is implemented internally using
pipes along with a fork/exec construct. The pipes are created, your
process automatically forks, and the child exec(2)'s the command you
wished to run.
This is all documented, and I thought I had a handle on this, but Perl's
open() function's error checking is based only on whether fork(2)
succeeded, not on whether exec(2) succeeded.
What I think we get when we don't check for errors is some kind of
raised signal (e.g. SIGPIPE) which is then duly received by the existing
async-IPC code (which is already doing multi-proc), which then kills the
IPC procs due to the 'impossible SIGPIPE' happening and crashes
kdesrc-build.
This fix, to check for an executable before trying to run, should at
least make this kind of error more apparent (this isn't perfect yet
either since 'bzr' failing to exist during pretend mode should be listed
as an update failure, not update success). I would still need to find a
better way to handle this since this is still a race condition.
CCBUG:345425
Trying to troubleshoot bug 345425, and I'm not sure if Perl might be
trying to handle data being transferred over the IPC pipes as Unicode. I
would think not but can find nothing in the docs that explicitly says
Perl won't do this, so force handling as binary just in case.
CCBUG:345425
This won't be released, at least not in this form. But it's time to put
what we have out there and update all those old websites, READMEs, and
Wikis. This is the first step in that.
Add and document the --include-dependencies option. Useful with
--print-modules as a quick way to determine what dependencies a given
module on the command line has, especially if you do a quick
"kdesrc-build --metadata-only" first.
Also, make selecting modules on the command line override
"include-dependencies" as an rc file option, since otherwise it's
impossible to ask just to build a single module if that module is in a
module set with included dependencies.
Use the newly-added --include-dependencies option to force dependency
resolution to happen even if you specify modules to build on the command
line.
Since now it may be possible to be searching for dependencies in this
code path even without having found a specific Module, we should pass
along the dependency name we're searching for so our recursive visitor.
Remove the special-case check on kf5umbrella by supporting all
virtual/misspelled dependencies in dependency-data without raising fatal
errors. --verbose still warns about missing dependencies (even with
kf5umbrella) and the worst case is that we try to build modules out of
order until someone notices the metadata is wrong.
That's a much better worst case than that of a couple of days ago
("kdesrc-build won't even start").
BUG:344814
So now I remember why kdesrc-build didn't recursively include
dependencies.... we took advantage of the fact that dependencies in
kdesrc-build were just for ordering (not for generating the build list)
by using 'kf5umbrella' as a fake module. This allowed workspace modules
to depend on only kf5umbrella instead of all of the frameworks
individually.
The recent changes to support recursive dependencies broke this, but I
wasn't able to reproduce until I introduce a module-set that actually
ran directly across kf5umbrella as a dependency, which is why I didn't
catch earlier.
This is only a workaround for kf5umbrella in particular, but should tide
things over until I can understand and fix it properly. Apologies for
the inconvenience.
This is a 2-part change:
1. Always download needed metadata, even in --pretend mode. This means
that even --pretend won't stop kdesrc-build from trying to hit the
network, but --pretend was always intended to let you test command lines
to ensure you got the right build list before you wasted hours trying to
build something. If you don't want kdesrc-build to touch the network
ever, don't run it. :)
2. Check for JSON decoding errors specifically (since it may happen that
the JSON is corrupted on download somehow, or accidentally broken
upstream).
This should allow "kdesrc-build -p --print-modules", or even
"kdesrc-build -p" to work even as the very first command ever run.
BUG:340481
FIXED-IN:1.16
E.g. Running "kdesrc-build kdeedu" with a kdesrc-buildrc containing
module-set
repository kde-projects
use-modules kdeedu
end module-set
would give a runtime exception, even though kdeedu expands to a valid
set of modules. The code here is too restrictive, looking for actual
specific *modules* (i.e. assuming that the use-modules entry specifies a
specific module instead of an implicit wildcard entry).
BUG:339965
FIXED-IN:1.16
There seems to be nothing better than checking for the existence of
files or directories under .git at this point (though it's possible to
decode git-status --porcelain output without too much trouble in the
case of in-progress merges).
This should hopefully help avoid kdesrc-build trying to update a module
that is being worked on.
BUG:334324
FIXED-IN:1.16
Bug 341159 complains about kdesrc-build-setup not setting up the kde:
git prefix, but kdesrc-build-setup itself never even calls git, it just
creates a .kdesrc-buildrc and exits.
However the currently recommended way to start out kdesrc-build is to
run "kdesrc-build --metadata-only" for the first build to setup
directories and download required data, while still letting a subsequent
"kdesrc-build --pretend" do something useful. kdesrc-build didn't setup
the kde: prefix in time to support --metadata-only, and I'm willing to
bet that's what the user was encountering here.
So, setup the git configuration before the first time we can use git.
BUG:341159
FIXED-IN:1.16
Even without include-dependencies, without this visit guard the tree
traversal code will add a ksb::Module dependency (when they are
encountered) to the build list every time it is encountered, not just
the first time it's encountered.
Currently it (probably) doesn't support pulling in options for these
modules from their host module-sets (although using 'options' sections
might actually work). I think it ignores the 'ignore-modules' option as
well.
Other than that, it should do pretty much what it says on the tin,
although I had to implement a few other bug fixes as well to get it to
that point.
I want to say that there was an open bug for this but I can't seem to
find it now.
Source items can never be catch-all/wildcard items, so we must
canonicalize these too to prevent e.g. extra-cmake-modules from
depending on kdesupport/extra-cmake-modules due to a '*:
kdesupport/extra-cmake-modules' rule.
The way kdesrc-build handles letting users type "use-modules frameworks"
and getting everything in frameworks/* back is by implementing the
wildcard search in the KDE XML reader that parses kde_projects.xml. This
uses a two-pass search for simple names, looking for 'name/*' and
'name'.
This results in duplicates, which are normally OK due to other features
in kdesrc-build that prevent duplicate modules from making it into the
build list. However a feature I'm implementing was made much more
difficult by this; easier instead just to prevent finding dups in the
first place.
This is somewhat confusing; since modules are listed in build order,
indented modules show up *before* the modules that require them.
Essentially, the module that caused an indented module to show up in its
position in the build is the next module in the build that is more
outdented.
Additionally, this only indents in situations where a module's build
order had to be rearranged. Modules that were already in the right order
are not specially indented even if a valid dependency exists.
With all that said, this can be a useful way of determining why
kdesrc-build thinks a given order is required without having to go the
route of a complicated ASCII DAG renderer.
This feature had been intended for cases where a module kde/foo was a
git repository while also being a base for other git repositories
kde/foo/bar, kde/foo/baz, etc. For instance, kde/kdelibs is a git
repository that also held sub-children like kde/kdelibs/kactivities.
However this is fairly rare in KDE repositories now, and this feature in
its easiest-to-implement form has always had to navigate around several
corner cases, so it's better just to explicitly specify dependencies.
This change also prefers the use of short module names. Additionally the
'inherent module dependencies' feature would likely be removed, although
I'm commiting the ported version here in case we'd ever need to bring it
back.
Fixes bug 342806, where a kdesrc-build user notes that "Cloning"
messages are indented while "Updating" messages are not, which can be
confusing.
BUG:342806
FIXED-IN:1.16
Right now kdesrc-build will default to kdesrc-buildrc-sample if no
configuration file is present, but that won't be appropriate now that
KF5 is out. Make the user make a choice instead.
The various READMEs and help texts would probably need to be updated to
match, if anyone wants to help. ;)
This commit makes the kde-build-metadata module a requirement instead of
an option, since the vast majority of runs will require this module now.
Additionally the --metadata-only option is added and documented to allow
for downloading the kde-build-metadata module alone (and make the
--pretend option work afterwards), and kdesrc-build recommends using it
if you run with --pretend and without metadata.
This should hopefully make the first-run use case easier for users.
BUG:337446
FIXED-IN:1.16
I had made some build-system related options not automatically apply to
modules with customized build systems (e.g. Qt, CMake, "custom"). But
this seems like the kind of option we'd want to have in common, as noted
in a bug report to me by Andre Wöbbeking, so make make-install-prefix
apply to all build systems by default if it's a global option.
This can, of course, still be overridden for individual modules or
module-sets.
Michael Pyne
Alex Richardson
Jeremy Whiting
Milian Wolff
Raphael Kubo da Costa