This still takes too long for most shell shortcuts, but will make it
possible to do things like setup shell functions to cd right to a build
dir based only on a module name, check for which git branch a module is
resolved to, or see what config file options a module ended up falling
under.
As part of the documentation changes, I also ported the docs to KF5.
JSON::PP is included in core Perl so one less dependency to look after,
and it's fast enough (for what we need) that there's really no point to
making it possible to look for JSON::XS.
This has the nice side effect of (finally) making KDEXMLReader free of
per-class global state (I keep forgetting Perl supports closures...),
but mostly this is to keep pace with the way we're using the source
repositories and grabbing potentially many dozens of modules at once
from the XML data. Better to just read it once and query rather than
parse anew for each and every set of queries.
The docs have been at docs.kde.org for a long time, now they're the one
and only recommended location. But I forgot to update the URLs
referenced by the script itself. :(
BUG:357162
FIXED-IN:16.01
As opposed to only kde-projects modules also present in the rc-file.
E.g. this rc-file
---
global
branch-group kf5-qt5
end global
module-set
repository kde-projects
use-modules plasma-desktop
include-dependencies true
end module-set
---
...still lists 83 modules to build if you run kdesrc-build.
We also needed to ensure that each module generated by user selectors
knows to propagate the 'needs dependencies' flag.
Commit 480f184 added a reference to the calligra module to get
lxr.kde.org to make calligra symbols available. However this also pulled
in other modules since kdesrc-build will normally take each entry in
use-modules as a virtual project path to dereference. For instance this
makes it possible to get all KDevelop-related modules by declaring
"use-modules kdevelop".
When this syntax is ambiguous (as it is for kdevelop and phonon) you
could normally do what you wanted by being more specific, using
declarations like "kdevelop/kdevelop" or "phonon/phonon".
However calligra.git is a top-level entry in the virtual project
hierarchy, so something like "/calligra" would still match calligra.git
(at /calligra) and krita (at /calligra/krita).
So, add Yet Another Syntax to allow for specifying a specific git
module while still taking advantage of the special kde-projects
niceties for module-sets. If you specify "foo.git" in a use-modules
entry, kdesrc-build will treat the module as if you'd asked for "foo",
but will not try to do any further searching for other modules in the
virtual project hierarchy.
CCMAIL:faure@kde.org
Now we describe up-to-date module progress (if multiple modules are
being built) and consolidate muliple lines of information into just a
couple (for build and install).
This option can be used to setup the user.name and user.email git-config
options within each new git repository cloned or installed from
snapshot, including kde-projects repositories. This can be useful for
KDE developers with multiple Git identities that they use for
development.
Module settings override global settings so you could e.g. have a global
git-user option for KDE modules and then override specific modules or
module-sets with a different git-user setting.
This setting is only applied during the initial git-clone process
though... if there's interest in having kdesrc-build re-apply settings
later that's possible too.
BUG:352478
FIXED-IN:15.09
The chomp doesn't return the chomp'ed value (rather '1' to reflect
number of characters removed), and fixing that revealed another bug with
my recent change to this code where I forgot to `tee' debug output to
the tty.
Should be a bit faster, but major goal here was to decouple the progress
output from the details of what we're outputting to. Functionality is
still the same (except that ninja progress output now works too) but now
it's that much easier to permit additional means of progress updates in
the future (e.g. a GUI, web server, even a Plasmoid).
When the initrepo script fails the extracted snapshot directory (the
srcdir) is deleted, but it's also the current working directory, which
breaks git when doing the fallback clone. Fix by switching to an
existing directory.
CCMAIL:faure@kde.org
Perl core includes a "HTTP::Tiny" module that does what little we need,
and has had it since 5.14 or so, so prefer that instead and remove the
requirement to install libwww-perl.
This also required removing the one minor instance of the URI module,
which is not technically in Perl core but was installed due to LWP.
Instead we just use a regexp 'blessed' by the URI module author.
There are currently various different paths to creating a Module (the
internal representation of a module from the rc-file). Depending on how
the Module is created you might get different options set for the Module
since the options are only available near the beginning (and are spread
across many different variables).
Consolidate some of this; this also has the side benefit of making
ksb::Application less of a dumping ground, and let's us get rid of a
'module-option-fixing callback' that needed to be used with a separate
'module-creating-by-name' callback.
Additionally this means that modules take only the options of the
module-set they were actually declared in, instead of the module-set in
use when the module was found (which may be different when using
--include-dependencies).
BUG:345192
FIXED-IN:15.05
I noticed that I was getting error messages from a git usage after
install on my KF5 modules, which are configured to use
remove-after-install.
Turns out that git or the shell or kernel or something doesn't like
running a command from a now-invalid cwd.
The recent(-ish) switch to allow short name from a few months ago in the
dependency-data files was buggy in the way it handles "catch all"
dependencies.
Dependency information was already shortened down to just the module
name when it was compared against the wildcard dependency rules, which
made it impossible for most of the rules to match (e.g. a dependency for
kde/kdemultimedia/* wouldn't match against 'juk' anymore because at that
portion of the code it's not clear that juk is actually a part of that
set).
I will still have to make some other fixes to permit wildcarded
dependencies to work through indirect modules. E.g. a dependency exists
for kde/foo/bar/bar* to "first-module". A second module depends on
"bar-intermedia-module", which is really under kde/foo/bar/. If
"bar-intermedia-module" is actually being built this is not an issue as
the dependency resolver can read the full path to detect the recursive
dependency onto "first-module". But if "bar-intermedia-module" is not
being built then there's no way to tell that there's really a dependency
on "first-module" since the wildcard dependency would not match.
Instead of requiring wildcards to immediately follow a '/' character in
branch-group handling (or to be a bare '*'), allow for wildcards
following other characters since it seems CI supports that syntax. This
should fix branch selection for the KDE Telepathy modules.
My last commit checked for missing executables before trying to execute
them, but still left open the possibility that it would be impossible to
call exec(2) on an executable that had just been found (for whatever
reason).
Closer inspection in 'perldoc perlipc', and 'perldoc -f open' has helped
illustrate how to catch that condition. E.g. it turns out that calling
'close' on a pipe-opened filehandle will automatically call waitpid,
plus I'd forgotten to exit in that error case in the child proc. (This
is why it would take 2-3 seconds after failing for the parent process to
continue).
The more interesting question is what error policy to use in this case.
Errors are expected in pretend mode (this function actually runs
processes even during pretend mode since otherwise it can be impossible
to determine whether to clone or update, switch branches, etc.), but on
the other hand during execution errors should be unexpected and
noteworthy.
FIXED-IN:15.04
BUG:345425
Perl's "open '-|' $program" feature, which is used to pipe the output of
a process to a filehandle variable, is implemented internally using
pipes along with a fork/exec construct. The pipes are created, your
process automatically forks, and the child exec(2)'s the command you
wished to run.
This is all documented, and I thought I had a handle on this, but Perl's
open() function's error checking is based only on whether fork(2)
succeeded, not on whether exec(2) succeeded.
What I think we get when we don't check for errors is some kind of
raised signal (e.g. SIGPIPE) which is then duly received by the existing
async-IPC code (which is already doing multi-proc), which then kills the
IPC procs due to the 'impossible SIGPIPE' happening and crashes
kdesrc-build.
This fix, to check for an executable before trying to run, should at
least make this kind of error more apparent (this isn't perfect yet
either since 'bzr' failing to exist during pretend mode should be listed
as an update failure, not update success). I would still need to find a
better way to handle this since this is still a race condition.
CCBUG:345425
Trying to troubleshoot bug 345425, and I'm not sure if Perl might be
trying to handle data being transferred over the IPC pipes as Unicode. I
would think not but can find nothing in the docs that explicitly says
Perl won't do this, so force handling as binary just in case.
CCBUG:345425
This won't be released, at least not in this form. But it's time to put
what we have out there and update all those old websites, READMEs, and
Wikis. This is the first step in that.
Add and document the --include-dependencies option. Useful with
--print-modules as a quick way to determine what dependencies a given
module on the command line has, especially if you do a quick
"kdesrc-build --metadata-only" first.
Also, make selecting modules on the command line override
"include-dependencies" as an rc file option, since otherwise it's
impossible to ask just to build a single module if that module is in a
module set with included dependencies.
Use the newly-added --include-dependencies option to force dependency
resolution to happen even if you specify modules to build on the command
line.
Michael Pyne
David Faure
Jeremy Whiting
Alex Richardson