I was so convinced this was already the case that I told dfaure that was
how the code worked and so it would be hard to find the bug he was
experienced. Whoops.
The corrected code now employs dependency resolution before the
second-pass --resume/--start handling, so that kdesrc-build should still
build modules after their dependencies are built.
This did give me the opportunity to clean up the metadataModule TODO
finally. Now there really is only one place that tracks whether we use
metadata or not, just like it should be.
The recent port to Getopt::Long broke the ability to use --foo=bar type
of options (where <foo> is a global option name known to the build
context). It also replaced the --module,foo=bar options with
--set-module-option-value=module,foo,bar.
While it's possible to use the latter syntax for global options (by
using 'global' as the module name) that is undocumented and may not work
forever. And either way, --foo options did use to work, and I noted it
was broken in the refactor.
So now that someone else has noticed I've reimplemented the feature.
It's difficult to do entirely from within Getopt::Long since the
documentation for that module gives you essentially one catch-all, which
only supports *non-options*.
So what I've done instead is to have kdesrc-build dynamically introspect
its list of global options and flags and then add them to the list of
valid options we pass to Getopt::Long. This also should mean that we
don't need to pass 'pass_through' as an option to Getopt::Long anymore,
though we'll see.
This code path won't be used for options that already have a command
line option (such as --async/--no-async) so that is still a misfeature
compared to how it was before, but I guess it can't be perfect. =D
I've tried to ensure that "flags" support the 'false' value to mean
boolean false, but be careful to pass false or 0 if that's what you
mean... specifying --dont-foo without giving it a value defaults it to
true.
BUG:330386
This is just a rearchitecture, but separates the internal build context
options into:
* flags (true/false settings), can be altered from cmdline.
* varible options, can be altered from cmdline.
* internal options, cannot be altered directly from cmdline.
The return result obtained from the waitpid($monitorPid) will account
for a failure of the update process and force $result to be 1 (an error
status).
If the update was successful then $result is unchanged, which means if
the build process had failed, then $result would still indicate failure.
That didn't work due to the addition of the 'my', which masked the
top-level $result.
BUG: 330388
Add a specific config file grouping (which acts just like a module
declaration), to allow for specifying options to override a
previously-declared module.
The use case here is for a module-set: You can specify options which
apply to an entire module-set when declaring the module-set, and then
override those options with any changes in a later "options"
declaration.
These declarations can stack too, so this can also be useful for
multi-level file includes (but this is less useful since an "options"
declaration requires a specific module, it doesn't work on module-sets;
in that case you'd want to have the different module-sets in your
most-specific included config files instead of in a base file).
Tested on my personal test case for bug 321883, and on a --pretend run,
and with a bug 321883 test case modified to not pre-declare the
overridden module first.
Example:
module-set kde-mm
repository kde-projects
use-modules kde/kdemultimedia
end module-set
options kmix # Not mentioned before this line
branch KDE/4.11
end options
In this case kmix would use KDE/4.11 branch while juk (and the rest of
kde-mm) would use whatever the global branch or branch-group was
(probably 'master').
BUG:321667
Proposed in http://lists.kde.org/?l=kde-commits&m=138887053920094,
though I am proposing that the common dependencies be stored in
"dependency-data-common" instead of "dependency-data-general".
In addition we would unilaterally choose a branch group now, falling
back to the existing 'use-stable-kde' option if needed.
Note that due to the way the code works, if you don't set
use-stable-kde, and don't set branch-group, you'll end up with
latest-qt4 for now. I would imagine this will change in the future
though!
As opposed to making them relative to script location, original rc-file,
initial cwd, or similar. This allows for whole directory trees to be
moved around if need be as well as long as the internal links remain
consistent.
I've tested this on a sample here and also on the provided test case
with the bug and it seems to work well although I have not had time to
run through a whole build, merely the --pretend sequence.
BUG:329444
FIXED-IN:1.16
Adds ignore-kde-structure option which will fetch and compile all
projects in the same dir, meaning that instead of:
extragear/network/kde-telepathy/ktp-text-ui
it would be just
ktp-text-ui
Handling options for the modules generated from a module-set has always been
especially clunky, and at some point even the hacks failed and it became
impossible to do things like setting an option for a specific module picked out
of a larger module-set.
E.g. the idea was always that the following would work:
module-set foo
repository kde-projects
use-modules kdelibs kde-workspace kde-runtime
cmake-options -DCMAKE_BUILD_TYPE=Release
end module-set
# Override the above, but only for kde-runtime
module kde-runtime
cmake-options -DCMAKE_BUILD_TYPE=Debug
end module
Probably I should have named "module" as "options" or even "override" (and
that's still feasible), but in any event this has been broken for awhile:
whenever kde-runtime is built it would end up with a Release build type instead
of a Debug one.
Worse yet, the second mention of kde-runtime was handled as a separate module.
Usually this duplicate module was weeded out by accident during the dependency
resolution phase, but that leaves open the question of which internal Module
object was the "winner"... now we can't even rely on the option handling being
predictably broken.
This was partially helped by c565d4c which at least prevents the code from
spitting out duplicate modules from within a given module-set (which is quite
easy to do by accident with kdelibs).
This commit reorganizes the command line and option reading code to do the
following:
- Add a "pending option" tracker, to hold option values that should be applied
to a named module, if one happens to be created, either via explicit
module-set expansion (e.g. if you ask for nepomuk-core or nepomuk-widgets) or
via implicit expansion (e.g. if you ask for kdemultimedia, you get juk and a
lot of others).
- Add a "selector" method which is responsible for translating module entries
on the command line into appropriate module-set or module selections from the
Modules and ModuleSets read in from the rc-file.
- Pass a subroutine to the module-set expansion method and the selector method
to ensure that any new Modules created as a part of either process are checked
for pending options (either from the rc-file due to the "override" module
method, or from the command line.
- While I was digging it out I made the long-overdue switch to Getopt::Long
away from my custom option parser. There are a couple of minor features lost in
this process but if they are needed I can add them back.
Now it should be possible to override options for individual modules within a
module-set. However as before, the module you wish to override *must* have been
mentioned somewhere before in a use-modules entry so that kdesrc-build can
recognize that it must hold onto those option values for later application. It
is still safe to mention a kde-projects module and its superset, they will be
re-ordered if necessary.
BUG:321883
FIXED-IN:1.16
These are usually stripped out later by the dependency analyzer but this
can't help the correctness of the code (e.g. with option handling, which
depends on there being only one right Module to apply options to).
By moving this code and associated utility methods into Application we
nearly complete the long-in-progress refactoring of the monolithic
kdesrc-build script into proper classes/objects.
The kdesrc-build script itself is nearly bare, but there's still a bit
more to do.
At least it should be easier to make individual test scripts to test
different bugs instead of having a monstrosity of a kdesrc-build-test
script that has to maintain a common state.
I've tested this in each way that I'm able (run needing to make a new
source dir, build dir, new install and log dir, run updating existing
checkouts, bzr/svn/git/kde-projects all tested, etc.). But this was a
large move so let me know if I've missed anything.
This is to support being able to improve and expand the test suite by
(eventually) having a simple function call to setup different modules to
be tested.
This is a long-overdue change that allows for overloading the
stringification operator to produce better error messages.
BuildException was previously not an actual module since kdesrc-build
itself was just a single script file. Now that we allow modules on disk
there is no reason to leave the exception class as a figment of Perl's
imagination.
Before this we would try to apply branch-group to a module, it would
fail (since plain git modules don't *have* any kde-projects metadata),
and we would invariably select master instead of choosing based on the
plain git module's branch option.
Note that this still doesn't solve the problem of modules that are from
kde-projects but *don't* have branch-group metadata; if both module
branch-group and global branch are active for such a module, the
module-specific branch-group will be tried and resolve to 'master'
independent of the global branch setting.
Whether this is a feature or not is not yet certain. :)
Now the priority order for all of the various "I want to checkout this
commit" options are centralized in one spot, and both cloning and
updating obey the same options now (they didn't before).
The spec allows for (or hints at, I forget) entries beginning with a "_"
in the layers and groups keys of the JSON object, so let's make sure we
filter those out just in case someone introduces one.
We need a better name for this feature.
But basically, this allows for creating kde-project repository-wide
"branch groups". For instance, you could list the git branch to use for
the "stable Qt/KDE 4" release/maintenance branch, the "new development
Qt/KDE 4" development branch (typically master, but that has been
changing), and the "upcoming next major release" branch (for us, this
means 'frameworks' sometimes, means 'master' sometimes, other modules
may have different ideas).
By using the 'branch-group' option (which remains to be further
documented) you can say that you want to defer to the metadata about the
repository branches stored in
kde-build-metadata/logical-module-structure. kdesrc-build should even
track branches for you as they are updated by the Release Team, if you
simply want to hang out on stable but compile your own stuff.
Note that this support requires Perl's JSON module (which should come
with Perl 5.14 if I understand it right, but you probably want to
install JSON::XS from CPAN). JSON support is only required for this
feature, it is not a new overall dependency.
See also: http://community.kde.org/Infrastructure/Project_Metadata
The ->isa() checks still work for superclasses, and knowing that a given
module originally came from a kde-project module set is useful in the
code. In fact this should be considered a regression from the module-set
parse refactoring patch.
This is unfortunately a giant change, as all of the functionality that
is encompassed into module-sets currently had to migrate over to
multiple separate classes, including the new ksb::ModuleSet class and
subclasses.
This was a long-overdue change, however, and should allow for accurately
tracking a source module-set for a given module.
On the other hand this migration of logic has made it easier to
understand each of the individual pieces where they stand (e.g. there is
no longer a separate expandXMLModules and expandModuleSets).
In addition we can properly handle ignore-modules with wildcards just as
we do with use-modules (they even use the same matching logic) which
means that it is safe to integrate this into master (assuming no extra
boogs get added, of course).
This will also help with fixing some of the extant module-selection bugs
(321883, 299415).
BUG:321275
CCBUG:321667
Kind of a mouthful, but remove a filter that was added to avoid
automatically including an actual git repository's children that were
also git repositories. E.g. think of something like a SuperBuild git
module that also had its component git modules as logical children
within projects.kde.org.
This filter is unnecessary now that kdesrc-build supports both kde.org
metadata (kde-build-metadata/build-script-ignore) and user-configurable
module filtering (ignore-modules option).
More importantly this filter precludes many other desirable types of
group syntax (e.g. including all of kdebindings just by asking for
kdebindings).
So we remove the filter. The trouble I've seen so far is that it is
now rather difficult to build *only* kdelibs, since kde/kdelibs is a
logical parent of nepomuk-core, nepomuk-widgets, and kactivities. This
can be worked-around by using a normal single module declaration. E.g.
module kdelibs
repository kde:kdelibs
end module
On the other hand, the recently-added config file option
"ignore-modules" should now remove any kde-projects modules that contain
a path component(s) matching the ignored module. You may have to be
specific with a given ignore atom for this reason (e.g. ignore
playground/libs, not 'libs' otherwise you'll also ignore
kdegraphics/libs at this point... though that was also true before this
commit).
BUG:321667
The previous version worked on modules that had *no* user (a setup most
of mine had as the user was set in the ssh config).
For modules with a user set the trailing '@' was included in the uid
which make the check fail even when the module was correct. (Though I
had thought I had tested this case with other modules).
In the past some KDE SVN modules used the "svn external" feature to pull
in other SVN repositories. These svn-external links were on the *server*
side so the link URL had to use a single scheme.
This scheme was https, which caused quite some problems for anonsvn
users just trying to update from SVN being an unattended kdesrc-build
run, as the SSL cert for svn.kde.org was not signed by a well-known CA.
As a result kdesrc-build updated svn automatically with the expected KDE
SVN SSL signature to avoid blocking on the interactive warning.
This is no longer required. The modules that used svn externals have
pretty much moved onto git now. The new svn server does not even have an
https interface, so this is dead code now, and removed accordingly.
Once I figure out a "preferred" solution for this use case I'll probably
convert to that, until then no reason to pollute with warnings.
I've run a search for other uses of given/when and didn't see any.
As opposed to reading /all/ output for a given module and only
displaying it immediately before the build phase starts. This makes it
very disconcerting to track git-clone progress...
Though this only works for Git modules, it should at least fix the
annoying "OMG YOUR REPO MOVED" warnings that never get fixed when in
async mode (which is the default).
Previously you simply never saw the warning due to the now-fixed output
squelching, now we need to fix this for good as well.
In synchronous mode it was impossible to get the build subroutine to
attempt to build a module which had no source update, as the updater
code would filter out its build phase *in addition* to marking it as
up-to-date.
This never broke the async code as filtering out the build phase was
always unimplemented (which is what led to a specific IPC type for "code
up-to-date" in the first place).
The build thread owns the TTY in the current setup. To keep the update
and build processess from fighting over the TTY when in async mode,
we've been simply squelching the update output up to this point, except
for errors.
This is not really the best idea as it can leave out kind of important
details. With this change we configure the logging subsystem to forward
its messages that it would ordinarily print out to instead go over an
IPC channel.
The IPC channel knows which module is currently being operated on, tags
the message, and forwards it over to destination proc's IPC handler,
which extracts it and holds onto the message until that proc reaches the
same module.
Once the receiver proc is processing the module, those messages are all
output at once and then normal processing resumes.
In case some messages are never read out this way (e.g. a module is
updated but never built) we have a message scavenger right after the
build is done to give those messages as well.
Michael Pyne
David E. Narvaez
Christoph Cullmann
Àlex Fiestas
David Faure
Martin Klapetek