sharpetronics/QGIS
1
0
Fork 0
mirror of https://github.com/qgis/QGIS.git synced 2025-02-22 00:06:12 -05:00
Code Issues Packages Projects Releases Wiki Activity

untwine pdalprovider integration (#40404)

Browse Source 
[pointclouds] untwine pdalprovider integration
This commit is contained in:
Peter Petrik 2020-12-15 14:25:09 +01:00 committed by GitHub
parent fe96e67741
commit 221bd2f68f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
70 changed files with 5377 additions and 85 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CMakeLists.txt
View File

@ -408,6 +408,9 @@ if(WITH_CORE)
endif()
if (WITH_PDAL)
if (NOT WITH_EPT)
message(FATAL_ERROR "PDAL provider cannot be built with EPT disabled")
endif()
find_package(PDAL) # PDAL provider
endif()
if (PDAL_FOUND)

674
external/untwine/LICENSE vendored Normal file
View File

@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.

54
external/untwine/api/QgisTest.cpp vendored Normal file
View File

@ -0,0 +1,54 @@
#ifndef _WIN32
#include <unistd.h>
#endif
#include <iostream>
#include "QgisUntwine.hpp"
int main()
{
untwine::QgisUntwine::StringList files;
untwine::QgisUntwine::Options options;
// std::string exe = "C:\\Users\\andre\\untwine\\build\\untwine.exe";
std::string exe = "/Users/acbell/untwine/build/untwine";
untwine::QgisUntwine api(exe);
// files.push_back("C:\\Users\\andre\\nyc2");
// files.push_back("C:\\Users\\andre\\nyc2\\18TXL075075.las.laz");
// files.push_back("/Users/acbell/nyc/18TXL075075.las.laz");
// files.push_back("/Users/acbell/nyc/18TXL075090.las.laz");
files.push_back("/Users/acbell/nyc2");
options.push_back({"dims", "X, Y, Z, Red, Green, Blue, Intensity"});
// book ok = api.start(files, ".\\out", options);
bool ok = api.start(files, "./out", options);
if (! ok)
{
std::cerr << "Couldn't start '" << exe << "!\n";
exit(-1);
}
bool stopped = false;
while (true)
{
#ifdef _WIN32
Sleep(1000);
#else
::sleep(1);
#endif
int percent = api.progressPercent();
std::string s = api.progressMessage();
std::cerr << "Percent/Msg = " << percent << " / " << s << "!\n";
/**
if (!stopped && percent >= 50)
{
stopped = true;
api.stop();
}
**/
if (!api.running())
break;
}
}

55
external/untwine/api/QgisUntwine.cpp vendored Normal file
View File

@ -0,0 +1,55 @@
#include <iostream>
#include "QgisUntwine.hpp"
#ifdef WIN32
#include "QgisUntwine_win.cpp"
#else
#include "QgisUntwine_unix.cpp"
#endif
namespace untwine
{
QgisUntwine::QgisUntwine(const std::string& untwinePath) : m_path(untwinePath), m_running(false),
m_percent(0)
{}
bool QgisUntwine::start(const StringList& files, const std::string& outputDir,
const Options& argOptions)
{
if (m_running)
return false;
Options options(argOptions);
if (files.size() == 0 || outputDir.empty())
return false;
std::string s;
for (auto ti = files.begin(); ti != files.end(); ++ti)
{
s += *ti;
if (ti + 1 != files.end())
s += ", ";
}
options.push_back({"files", s});
options.push_back({"output_dir", outputDir});
return start(options);
}
int QgisUntwine::progressPercent() const
{
readPipe();
return m_percent;
}
std::string QgisUntwine::progressMessage() const
{
readPipe();
return m_progressMsg;
}
} // namespace untwine

48
external/untwine/api/QgisUntwine.hpp vendored Normal file
View File

@ -0,0 +1,48 @@
#pragma once
#include <map>
#include <string>
#include <vector>
#ifdef _WIN32
#include <Windows.h>
#endif
namespace untwine
{
class QgisUntwine
{
public:
using Option = std::pair<std::string, std::string>;
using Options = std::vector<Option>;
using StringList = std::vector<std::string>;
QgisUntwine(const std::string& untwinePath);
bool start(const StringList& files, const std::string& outputDir,
const Options& argOptions = Options());
bool stop();
bool running();
int progressPercent() const;
std::string progressMessage() const;
private:
std::string m_path;
mutable bool m_running;
mutable int m_percent;
mutable std::string m_progressMsg;
#ifndef _WIN32
pid_t m_pid;
int m_progressFd;
#else
HANDLE m_pid;
HANDLE m_progressFd;
#endif
bool start(Options& options);
void readPipe() const;
void childStopped();
};
} // namespace untwine

141
external/untwine/api/QgisUntwine_unix.cpp vendored Normal file
View File

@ -0,0 +1,141 @@
#include <iostream>
#include <fcntl.h>
#include <signal.h>
#include <unistd.h>
#include <sys/errno.h>
#include <sys/wait.h>
#include "QgisUntwine.hpp"
namespace untwine
{
bool QgisUntwine::start(Options& options)
{
int fd[2];
int ret = ::pipe(fd);
m_pid = ::fork();
// Child
if (m_pid == 0)
{
// Close file descriptors other than the stdin/out/err and our pipe.
// There may be more open than FD_SETSIZE, but meh.
for (int i = STDERR_FILENO + 1; i < FD_SETSIZE; ++i)
if (i != fd[1])
close(i);
// Add the FD for the progress output
options.push_back({"progress_fd", std::to_string(fd[1])});
for (Option& op : options)
op.first = "--" + op.first;
std::vector<const char *> argv;
argv.push_back(m_path.data());
for (const Option& op : options)
{
argv.push_back(op.first.data());
argv.push_back(op.second.data());
}
argv.push_back(nullptr);
if (::execv(m_path.data(), const_cast<char *const *>(argv.data())) != 0)
{
std::cerr << "Couldn't start untwine '" << m_path << "'.\n";
exit(-1);
}
}
// Parent
else
{
close(fd[1]);
m_progressFd = fd[0];
// Don't block attempting to read progress.
::fcntl(m_progressFd, F_SETFL, O_NONBLOCK);
m_running = true;
}
return true;
}
bool QgisUntwine::stop()
{
if (!m_running)
return false;
::kill(m_pid, SIGINT);
(void)waitpid(m_pid, nullptr, 0);
m_pid = 0;
return true;
}
// Called when the child has stopped.
void QgisUntwine::childStopped()
{
m_running = false;
}
bool QgisUntwine::running()
{
if (m_running && (::waitpid(m_pid, nullptr, WNOHANG) != 0))
childStopped();
return m_running;
}
namespace
{
uint32_t readString(int fd, std::string& s)
{
uint32_t ssize;
// Loop while there's nothing to read. Generally this shouldn't loop.
while (true)
{
ssize_t numRead = read(fd, &ssize, sizeof(ssize));
if (numRead == -1 && errno != EAGAIN)
continue;
else if (numRead == sizeof(ssize))
break;
else
return -1; // Shouldn't happen.
}
// Loop reading string
char buf[80];
std::string t;
while (ssize)
{
ssize_t toRead = (std::min)((size_t)ssize, sizeof(buf));
ssize_t numRead = read(fd, buf, toRead);
if (numRead == 0 || (numRead == -1 && errno != EAGAIN))
return -1;
if (numRead > 0)
{
ssize -= numRead;
t += std::string(buf, numRead);
}
}
s = std::move(t);
return 0;
}
} // unnamed namespace
void QgisUntwine::readPipe() const
{
// Read messages until the pipe has been drained.
while (true)
{
ssize_t size = read(m_progressFd, &m_percent, sizeof(m_percent));
// If we didn't read the full size, just return.
if (size != sizeof(m_percent))
return;
// Read the string, waiting as necessary.
if (readString(m_progressFd, m_progressMsg) != 0)
break;
}
}
} // namespace untwine

148
external/untwine/api/QgisUntwine_win.cpp vendored Normal file
View File

@ -0,0 +1,148 @@
#include <iostream>
#include "QgisUntwine.hpp"
namespace untwine
{
bool QgisUntwine::start(Options& options)
{
HANDLE handle[2];
SECURITY_ATTRIBUTES pipeAttr;
pipeAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
pipeAttr.bInheritHandle = TRUE;
pipeAttr.lpSecurityDescriptor = NULL;
CreatePipe(&handle[0], &handle[1], &pipeAttr, 0);
// Set the read end to no-wait.
DWORD mode = PIPE_NOWAIT;
SetNamedPipeHandleState(handle[0], &mode, NULL, NULL);
size_t xhandle = reinterpret_cast<size_t>(handle[1]);
options.push_back({"progress_fd", std::to_string(xhandle)});
std::string cmdline;
cmdline += m_path + " ";
for (const Option& op : options)
cmdline += "--" + op.first + " \"" + op.second + "\" ";
PROCESS_INFORMATION processInfo;
STARTUPINFO startupInfo;
ZeroMemory(&processInfo, sizeof(PROCESS_INFORMATION));
ZeroMemory(&startupInfo, sizeof(STARTUPINFO));
startupInfo.cb = sizeof(STARTUPINFO);
/**
startupInfo.hStdInput = GetStdHandle(STD_INPUT_HANDLE);
startupInfo.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
startupInfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
startupInfo.dwFlags = STARTF_USESTDHANDLES;
**/
std::vector<char> ncCmdline(cmdline.begin(), cmdline.end());
ncCmdline.push_back((char)0);
bool ok = CreateProcessA(m_path.c_str(), ncCmdline.data(),
NULL, /* process attributes */
NULL, /* thread attributes */
TRUE, /* inherit handles */
CREATE_NO_WINDOW, /* creation flags */
NULL, /* environment */
NULL, /* current directory */
&startupInfo, /* startup info */
&processInfo /* process information */
);
if (ok)
{
m_pid = processInfo.hProcess;
m_progressFd = handle[0];
m_running = true;
}
return ok;
}
bool QgisUntwine::stop()
{
if (!m_running)
return false;
TerminateProcess(m_pid, 1);
WaitForSingleObject(m_pid, INFINITE);
childStopped();
m_pid = 0;
return true;
}
// Called when the child has stopped.
void QgisUntwine::childStopped()
{
m_running = false;
CloseHandle(m_progressFd);
CloseHandle(m_pid);
}
bool QgisUntwine::running()
{
if (m_running && WaitForSingleObject(m_pid, 0) != WAIT_TIMEOUT)
childStopped();
return m_running;
}
namespace
{
int readString(HANDLE h, std::string& s)
{
uint32_t ssize;
// Loop while there's nothing to read. Generally this shouldn't loop.
while (true)
{
DWORD numRead;
bool ok = ReadFile(h, &ssize, sizeof(ssize), &numRead, NULL);
// EOF or nothing to read.
if (numRead == 0 && GetLastError() == ERROR_NO_DATA)
continue;
else if (numRead == sizeof(ssize))
break;
else
return -1;
}
// Loop reading string
char buf[80];
std::string t;
while (ssize)
{
DWORD numRead;
DWORD toRead = (std::min)((size_t)ssize, sizeof(buf));
ReadFile(h, buf, toRead, &numRead, NULL);
if (numRead == 0 && GetLastError() == ERROR_NO_DATA)
continue;
if (numRead <= 0)
return -1;
ssize -= numRead;
t += std::string(buf, numRead);
}
s = std::move(t);
return 0;
}
} // unnamed namespace
void QgisUntwine::readPipe() const
{
// Read messages until the pipe has been drained.
while (true)
{
DWORD numRead;
ReadFile(m_progressFd, &m_percent, sizeof(m_percent), &numRead, NULL);
if (numRead != sizeof(m_percent))
return;
// Read the string, waiting as necessary.
if (readString(m_progressFd, m_progressMsg) != 0)
break;
}
}
} // namespace untwine

45
external/untwine/api/README vendored Normal file
View File

@ -0,0 +1,45 @@
This interface is a process-level API. This means that only a thin set of code is linked to
yours. Untwine is started as an independent process by this API. Functions are provided to
control the Untwine process that is started by this API. Running Untwine as a separate process
has the advantage that if there are errors or bugs in Untwine or the program fails in any way,
the user of this API is unaffected.
QgisUntwine(const std::string& untwinePath)
Initialize the Untwine API to run the untwine executable at the 'untwinePath'.
bool start(const StringList& files, const std::string& outputDir, const Options& argOptions)
Start Untwine to ingest 'files' and write the output to 'outputDir'. Options
can be optionally passed to Untwine via the 'argOptions' argument. Returns
true if Untwine is successfully started, false otherwise.
bool stop()
Stop the running Untwine process. Returns true if Untwine was running and an attempt
to stop was made, false otherwise.
bool running()
Returns true if Untwine is running, false otherwise.
int progressPercent()
Returns the approximate percentage (0-100) of completion last reported by the Untwine
application.
std::string progressMessage()
Returns the most recent progress message reported by the Untwine application.
API Usage Notes
---------------
If Untwine completes successfully, you must call running() or stop() in order for your
process to properly clean-up after the untwine process. If your process terminates before
Untwine completes, Untwine will normally see that the pipe that connects your process and
Untwine was closed and will exit, though it may take some small amount of time for this to occur.
If you want to make sure that a failure of your process (like a segmentation fault) terminates
the Untwine process immediately, you must call stop() from the appropriate signal handler,
though, again, this is generally unnecessary.

277
external/untwine/bu/BuPyramid.cpp vendored Normal file
View File

@ -0,0 +1,277 @@
#include <iomanip>
#include <regex>
#include <set>
#include <string>
#include <vector>
#include <pdal/util/ProgramArgs.hpp>
#include "BuPyramid.hpp"
#include "BuTypes.hpp"
#include "FileInfo.hpp"
#include "OctantInfo.hpp"
#include "../untwine/Common.hpp"
#include "../untwine/ProgressWriter.hpp"
namespace untwine
{
namespace bu
{
/// BuPyramid
BuPyramid::BuPyramid() : m_manager(m_b)
{}
void BuPyramid::run(const Options& options, ProgressWriter& progress)
{
m_b.inputDir = options.tempDir;
m_b.outputDir = options.outputDir;
readBaseInfo();
getInputFiles();
size_t count = queueWork();
progress.setPercent(.6);
progress.setIncrement(.4 / count);
m_manager.setProgress(&progress);
std::thread runner(&PyramidManager::run, &m_manager);
runner.join();
writeInfo();
}
void BuPyramid::readBaseInfo()
{
auto nextblock = [](std::istream& in)
{
std::string s;
bool firstnl = false;
while (in)
{
char c;
in.get(c);
if (c == '\n')
{
if (firstnl)
{
// Remove trailing newline.
s.resize(s.size() - 1);
return s;
}
else
firstnl = true;
}
else
firstnl = false;
s += c;
}
return s;
};
std::string baseFilename = m_b.inputDir + "/" + MetadataFilename;
std::ifstream in(baseFilename);
if (!in)
fatal("Can't open '" + MetadataFilename + "' in directory '" + m_b.inputDir + "'.");
std::stringstream ss(nextblock(in));
ss >> m_b.bounds.minx >> m_b.bounds.miny >> m_b.bounds.minz;
ss >> m_b.bounds.maxx >> m_b.bounds.maxy >> m_b.bounds.maxz;
ss.str(nextblock(in));
ss.clear();
ss >> m_b.trueBounds.minx >> m_b.trueBounds.miny >> m_b.trueBounds.minz;
ss >> m_b.trueBounds.maxx >> m_b.trueBounds.maxy >> m_b.trueBounds.maxz;
std::string srs = nextblock(in);
if (srs != "NONE")
m_b.srs.set(srs);
if (!in)
throw "Couldn't read info file.";
ss.str(nextblock(in));
ss.clear();
m_b.pointSize = 0;
while (true)
{
FileDimInfo fdi;
ss >> fdi;
if (!ss)
break;
if (fdi.name.empty())
fatal("Invalid dimension in info.txt.");
m_b.pointSize += pdal::Dimension::size(fdi.type);
m_b.dimInfo.push_back(fdi);
}
if (m_b.pointSize == 0)
throw "Couldn't read info file.";
}
void BuPyramid::writeInfo()
{
auto typeString = [](pdal::Dimension::BaseType b)
{
using namespace pdal::Dimension;
switch (b)
{
case BaseType::Signed:
return "signed";
case BaseType::Unsigned:
return "unsigned";
case BaseType::Floating:
return "float";
default:
return "";
}
};
std::ofstream out(m_b.outputDir + "/ept.json");
out << "{\n";
pdal::BOX3D& b = m_b.bounds;
std::ios init(NULL);
init.copyfmt(out);
out << std::fixed << std::setw(12);
out << "\"bounds\": [" <<
b.minx << ", " << b.miny << ", " << b.minz << ", " <<
b.maxx << ", " << b.maxy << ", " << b.maxz << "],\n";
pdal::BOX3D& tb = m_b.trueBounds;
out << "\"boundsConforming\": [" <<
tb.minx << ", " << tb.miny << ", " << tb.minz << ", " <<
tb.maxx << ", " << tb.maxy << ", " << tb.maxz << "],\n";
out << "\"dataType\": \"laszip\",\n";
out << "\"hierarchyType\": \"json\",\n";
out << "\"points\": " << m_manager.totalPoints() << ",\n";
out << "\"span\": 128,\n";
out << "\"version\": \"1.0.0\",\n";
out << "\"schema\": [\n";
for (auto di = m_b.dimInfo.begin(); di != m_b.dimInfo.end(); ++di)
{
const FileDimInfo& fdi = *di;
out << "\t{";
out << "\"name\": \"" << fdi.name << "\", ";
out << "\"type\": \"" << typeString(pdal::Dimension::base(fdi.type)) << "\", ";
if (fdi.name == "X" || fdi.name == "Y" || fdi.name == "Z")
out << "\"scale\": 0.01, \"offset\": 0, ";
out << "\"size\": " << pdal::Dimension::size(fdi.type) << " ";
out << "}";
if (di + 1 != m_b.dimInfo.end())
out << ",";
out << "\n";
}
out << "],\n";
out << "\"srs\": {\n";
if (m_b.srs.valid())
{
out << "\"wkt\": " << "\"" << pdal::Utils::escapeJSON(m_b.srs.getWKT()) << "\"\n";
}
out << "}\n";
out << "}\n";
out.copyfmt(init);
}
void BuPyramid::getInputFiles()
{
auto matches = [](const std::string& f)
{
std::regex check("([0-9]+)-([0-9]+)-([0-9]+)-([0-9]+)\\.bin");
std::smatch m;
if (!std::regex_match(f, m, check))
return std::make_pair(false, VoxelKey(0, 0, 0, 0));
int level = std::stoi(m[1].str());
int x = std::stoi(m[2].str());
int y = std::stoi(m[3].str());
int z = std::stoi(m[4].str());
return std::make_pair(true, VoxelKey(x, y, z, level));
};
std::vector<std::string> files = pdal::FileUtils::directoryList(m_b.inputDir);
VoxelKey root;
for (std::string file : files)
{
uintmax_t size = pdal::FileUtils::fileSize(file);
file = pdal::FileUtils::getFilename(file);
auto ret = matches(file);
bool success = ret.first;
VoxelKey& key = ret.second;
if (success)
m_allFiles.emplace(key, FileInfo(file, size / m_b.pointSize));
}
// Remove any files that are hangers-on from a previous run - those that are parents
// of other input.
for (auto it = m_allFiles.begin(); it != m_allFiles.end(); ++it)
{
VoxelKey k = it->first;
while (k != root)
{
k = k.parent();
m_allFiles.erase(k);
};
}
}
size_t BuPyramid::queueWork()
{
std::set<VoxelKey> needed;
std::set<VoxelKey> parentsToProcess;
std::vector<OctantInfo> have;
const VoxelKey root;
for (auto& afi : m_allFiles)
{
VoxelKey k = afi.first;
FileInfo& f = afi.second;
// Stick an OctantInfo for this file in the 'have' list.
OctantInfo o(k);
o.appendFileInfo(f);
have.push_back(o);
// Walk up the tree and make sure that we're populated for all children necessary
// to process to the top level.
while (k != root)
{
k = k.parent();
parentsToProcess.insert(k);
for (int i = 0; i < 8; ++i)
needed.insert(k.child(i));
}
}
// Now remove entries for the files we have and their parents.
for (const OctantInfo& o : have)
{
VoxelKey k = o.key();
while (k != root)
{
needed.erase(k);
k = k.parent();
}
}
// Queue what we have and what's left that's needed.
for (const OctantInfo& o : have)
m_manager.queue(o);
for (const VoxelKey& k : needed)
m_manager.queue(OctantInfo(k));
return parentsToProcess.size();
}
} // namespace bu
} // namespace untwine

57
external/untwine/bu/BuPyramid.hpp vendored Normal file
View File

@ -0,0 +1,57 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <string>
#include <unordered_map>
#include <vector>
#include "PyramidManager.hpp"
#include "BuTypes.hpp"
namespace pdal
{
class ProgramArgs;
}
namespace untwine
{
struct Options;
class ProgressWriter;
namespace bu
{
class FileInfo;
class BuPyramid
{
public:
BuPyramid();
void run(const Options& options, ProgressWriter& progress);
private:
void getInputFiles();
void readBaseInfo();
size_t queueWork();
void writeInfo();
PyramidManager m_manager;
BaseInfo m_b;
std::unordered_map<VoxelKey, FileInfo> m_allFiles;
};
} // namespace bu
} // namespace untwine

41
external/untwine/bu/BuTypes.hpp vendored Normal file
View File

@ -0,0 +1,41 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <stdexcept>
#include <pdal/SpatialReference.hpp>
#include <pdal/util/Bounds.hpp>
#include "../untwine/FileDimInfo.hpp"
namespace untwine
{
namespace bu
{
struct BaseInfo
{
pdal::BOX3D bounds;
pdal::BOX3D trueBounds;
size_t pointSize;
std::string inputDir;
std::string outputDir;
int maxLevel;
DimInfoList dimInfo;
pdal::SpatialReference srs;
};
} // namespace bu
} // namespace untwine

61
external/untwine/bu/FileInfo.hpp vendored Normal file
View File

@ -0,0 +1,61 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <list>
#include <pdal/util/FileUtils.hpp>
namespace untwine
{
namespace bu
{
class FileInfo
{
public:
FileInfo(const std::string& filename, size_t numPoints) :
m_filename(filename), m_numPoints(numPoints)
{}
std::string filename() const
{ return m_filename; }
int numPoints() const
{ return m_numPoints; }
// When sampling we give a single set of indices to the points in the various
// input files. m_start is the starting index of the points in this file.
void setStart(int start)
{ m_start = start; }
int start() const
{ return m_start; }
char *address() const
{ return reinterpret_cast<char *>(m_ctx.addr()); }
pdal::FileUtils::MapContext context() const
{ return m_ctx; }
void setContext(const pdal::FileUtils::MapContext& ctx)
{ m_ctx = ctx; }
private:
std::string m_filename;
int m_numPoints;
int m_start;
pdal::FileUtils::MapContext m_ctx;
};
using FileInfoList = std::list<FileInfo>;
} // namespace bu
} // namespace untwine

79
external/untwine/bu/OctantInfo.hpp vendored Normal file
View File

@ -0,0 +1,79 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include "../untwine/VoxelKey.hpp"
#include "FileInfo.hpp"
namespace untwine
{
namespace bu
{
class OctantInfo
{
public:
OctantInfo() : m_mustWrite(false)
{}
OctantInfo(const VoxelKey& key) : m_mustWrite(false)
{ m_key = key; }
void appendFileInfo(const FileInfo& fi)
{ m_fileInfos.push_back(fi); }
void prependFileInfo(const FileInfo& fi)
{ m_fileInfos.push_front(fi); }
void appendFileInfos(OctantInfo& o)
{ m_fileInfos.splice(m_fileInfos.end(), o.m_fileInfos); }
size_t numPoints() const
{
size_t cnt = 0;
for (const FileInfo& fi : m_fileInfos)
cnt += fi.numPoints();
return cnt;
}
bool hasPoints() const
{
for (const FileInfo& fi : m_fileInfos)
if (fi.numPoints())
return true;
return false;
}
std::list<FileInfo>& fileInfos()
{ return m_fileInfos; }
const std::list<FileInfo>& fileInfos() const
{ return m_fileInfos; }
VoxelKey key() const
{ return m_key; }
void setKey(VoxelKey k)
{ m_key = k; }
bool mustWrite() const
{ return m_mustWrite; }
void setMustWrite(bool mustWrite)
{ m_mustWrite = mustWrite; }
private:
FileInfoList m_fileInfos;
VoxelKey m_key;
bool m_mustWrite;
};
} // namespace bu
} // namespace untwine

82
external/untwine/bu/PointAccessor.hpp vendored Normal file
View File

@ -0,0 +1,82 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include "../untwine/Common.hpp"
#include "../untwine/Point.hpp"
#include "BuTypes.hpp"
#include "FileInfo.hpp"
namespace untwine
{
namespace bu
{
class PointAccessor
{
public:
PointAccessor(const BaseInfo& b) : m_b(b)
{}
~PointAccessor()
{
for (FileInfo *fi : m_fileInfos)
pdal::FileUtils::unmapFile(fi->context());
}
void read(FileInfo& fi)
{
using namespace pdal::FileUtils;
std::string filename = m_b.inputDir + "/" + fi.filename();
auto ctx = mapFile(filename, true, 0, fi.numPoints() * m_b.pointSize);
if (ctx.m_addr == nullptr)
fatal(filename + ": " + ctx.m_error);
fi.setContext(ctx);
fi.setStart(size());
m_fileInfos.push_back(&fi);
}
Point operator[](size_t offset)
{
for (FileInfo *fi : m_fileInfos)
if (offset >= (size_t)fi->start() &&
offset < (size_t)fi->start() + fi->numPoints())
return Point(fi->address() + ((offset - fi->start()) * m_b.pointSize));
return Point();
}
size_t size()
{
if (m_fileInfos.empty())
return 0;
else
return m_fileInfos.back()->start() + m_fileInfos.back()->numPoints();
}
void dump()
{
std::cerr << "Accessor infos:\n";
for (FileInfo *fi : m_fileInfos)
std::cerr << fi->filename() << "/" << fi->start() << "/" << fi->numPoints() << "!\n";
std::cerr << "\n";
}
private:
const BaseInfo& m_b;
std::vector<FileInfo *> m_fileInfos;
};
} // namespace bu
} // namespace untwine

371
external/untwine/bu/Processor.cpp vendored Normal file
View File

@ -0,0 +1,371 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <numeric>
#include <random>
#include "../untwine/GridKey.hpp"
#include <pdal/StageFactory.hpp>
#include <pdal/io/BufferReader.hpp>
#include "Processor.hpp"
#include "PyramidManager.hpp"
namespace untwine
{
namespace bu
{
static const int MinimumPoints = 100;
static const int MinimumTotalPoints = 1500;
Processor::Processor(PyramidManager& manager, const VoxelInfo& v, const BaseInfo& b) :
m_vi(v), m_b(b), m_manager(manager), m_points(m_b)
{}
void Processor::run()
{
size_t totalPoints = 0;
size_t totalFileInfos = 0;
for (int i = 0; i < 8; ++i)
{
OctantInfo& child = m_vi[i];
totalFileInfos += child.fileInfos().size();
totalPoints += child.numPoints();
if (child.numPoints() < MinimumPoints)
m_vi.octant().appendFileInfos(child);
}
// It's possible that all the file infos have been moved above, but this is cheap.
if (totalPoints < MinimumTotalPoints)
for (int i = 0; i < 8; ++i)
m_vi.octant().appendFileInfos(m_vi[i]);
// Accepted points are those that will go in this (the parent) cell.
// Rejected points will remain in the child cell they were in previously.
Index accepted;
Index rejected;
// If the file infos haven't all been hoisted, sample.
if (m_vi.octant().fileInfos().size() != totalFileInfos)
sample(accepted, rejected);
write(accepted, rejected);
m_manager.queue(m_vi.octant());
}
void Processor::sample(Index& accepted, Index& rejected)
{
int totalPoints = 0;
for (int i = 0; i < 8; ++i)
{
OctantInfo& child = m_vi[i];
for (FileInfo& fi : child.fileInfos())
{
m_points.read(fi);
totalPoints += fi.numPoints();
}
}
std::deque<int> index(totalPoints);
std::iota(index.begin(), index.end(), 0);
std::random_device rd;
std::mt19937 g(rd());
//ABELL - This may not be the best way to do this. Probably better to work from some
// point (center, whatever) out, but this is cheap because you don't have to do
// any computation with the point data. And I would think you should still get good
// output, but it may be more sparse. Seems you could fix that by just choosing a
// smaller radius. Should be tested.
std::shuffle(index.begin(), index.end(), g);
while (index.size())
{
int i = index.back();
const Point& p = m_points[i];
GridKey k = m_vi.gridKey(p);
// If we're accepting this point into this voxel from it's child, add it
// to the accepted list and also stick it in the grid.
if (acceptable(i, k))
{
accepted.push_back(i);
m_vi.grid().insert( {k, i} );
}
else
rejected.push_back(i);
index.pop_back();
}
}
void Processor::write(Index& accepted, Index& rejected)
{
/**
std::cerr << m_vi.key() << " Accepted/Rejected/num points = " <<
accepted.size() << "/" << rejected.size() << "/" << m_vi.numPoints() << "!\n";
**/
// If this is the final key, append any remaining file infos as accepted points and
// write the accepted points as binary.
if (m_vi.key() == VoxelKey(0, 0, 0, 0))
{
appendRemainder(accepted);
writeOctantCompressed(m_vi.octant(), accepted, accepted.begin());
}
else
writeBinOutput(accepted);
writeCompressedOutput(rejected);
}
bool Processor::acceptable(int pointId, GridKey key)
{
VoxelInfo::Grid& grid = m_vi.grid();
auto it = grid.find(key);
// If the cell is already occupied the point is not acceptable.
if (it != grid.end())
return false;
return true;
/**
// We place points in a voxel grid to reduce the number of tests necessary
// to determine if a new point can be placed without being too close.
// We size the voxels such that the diagonal is the length of our radius.
// This way we KNOW that once a cell is occupied, no other point can
// be placed there. This means the edge length of the voxel cell is
// radius / √3 = .577 * radius. So, when trying to place a point on the far
// right side of a cell, it's possible that there's another point already in
// a cell 2 cells to the right that's only radius * .577 + ε away.
// ABELL - This should probably be moved to a Grid class.
// Ignore cells outside of the area of interest.
int i0 = std::max(key.i() - 2, 0);
int j0 = std::max(key.j() - 2, 0);
int k0 = std::max(key.k() - 2, 0);
int i1 = std::min(key.i() + 2, m_vi.gridXCount());
int j1 = std::min(key.j() + 2, m_vi.gridYCount());
int k1 = std::min(key.k() + 2, m_vi.gridZCount());
for (int i = i0; i <= i1; ++i)
for (int j = j0; j <= j1; ++j)
for (int k = k0; k <= k1; ++k)
{
//ABELL - Is it worth skipping key location itself or the corner cells?
auto gi = grid.find(GridKey(i, j, k));
if (gi != grid.end() && tooClose(pointId, gi->second))
return false;
}
return true;
**/
}
bool Processor::tooClose(pdal::PointId id1, pdal::PointId id2)
{
const Point& p1 = m_points[id1];
const Point& p2 = m_points[id2];
double dx = p1.x() - p2.x();
double dy = p1.y() - p2.y();
double dz = p1.z() - p2.z();
return dx * dx + dy * dy + dz * dz <= m_vi.squareSpacing();
}
void Processor::writeBinOutput(Index& index)
{
if (index.empty())
return;
// Write the accepted points in binary format. Create a FileInfo to describe the
// file and it to the octant representing this voxel as it bubbles up.
// Note that we write the the input directory, as this will be input to a later
// pass.
std::string filename = m_vi.key().toString() + ".bin";
std::string fullFilename = m_b.inputDir + "/" + filename;
std::ofstream out(fullFilename, std::ios::binary | std::ios::trunc);
if (!out)
fatal("Couldn't open '" + fullFilename + "' for output.");
for (size_t i = 0; i < index.size(); ++i)
out.write(m_points[index[i]].cdata(), m_b.pointSize);
m_vi.octant().appendFileInfo(FileInfo(filename, index.size()));
}
// This is a bit confusing. When we get to the last node, we have two sets of points that
// need to get written to the final (0-0-0-0) node. Those include the points accepted from
// sampling as well as any points that were simply hoisted here due to small size.
//
// We read the hoisted points to stick them on the PointAccessor then we number the points
// by adding them to the index (accepted list). Then we move the FileInfos from the
void Processor::appendRemainder(Index& index)
{
std::sort(index.begin(), index.end());
// Save the current size;
size_t offset = m_points.size();
// Read the points from the remaining FileInfos.
for (FileInfo& fi : m_vi.octant().fileInfos())
m_points.read(fi);
size_t numRead = m_points.size() - offset;
size_t origIndexSize = index.size();
// Resize the index to contain the read points.
index.resize(origIndexSize + numRead);
// The points in the remaining hoisted FileInfos are just numbered sequentially.
std::iota(index.begin() + origIndexSize, index.end(), offset);
// NOTE: We need to maintain the order of the file infos as they were read, which
// is why they're prepended in reverse.
// NOTE: The FileInfo pointers in the PointAccessor should remain valid as the
// file infos are spliced from the child octant lists onto the parent list.
for (int i = 8; i > 0; i--)
{
FileInfoList& fil = m_vi[i - 1].fileInfos();
for (auto fi = fil.rbegin(); fi != fil.rend(); ++fi)
m_vi.octant().prependFileInfo(*fi);
}
}
void Processor::writeCompressedOutput(Index& index)
{
// By sorting the rejected points, they will be ordered to match the FileInfo items --
// meaning that all points that belong in one file will be consecutive.
std::sort(index.begin(), index.end());
IndexIter pos = index.begin();
for (int octant = 0; octant < 8; ++octant)
if (m_vi[octant].hasPoints() || m_vi[octant].mustWrite())
{
m_vi.octant().setMustWrite(true);
pos = writeOctantCompressed(m_vi[octant], index, pos);
}
}
Processor::IndexIter
Processor::writeOctantCompressed(const OctantInfo& o, Index& index, IndexIter pos)
{
auto begin = pos;
pdal::PointTable table;
//ABELL - fixme
// For now we copy the dimension list so we're sure that it matches the layout, though
// there's no reason why it should change. We should modify things to use a single
// layout.
DimInfoList dims = m_b.dimInfo;
for (FileDimInfo& fdi : dims)
fdi.dim = table.layout()->registerOrAssignDim(fdi.name, fdi.type);
table.finalize();
pdal::PointViewPtr view(new pdal::PointView(table));
auto fii = o.fileInfos().begin();
auto fiiEnd = o.fileInfos().end();
size_t count = 0;
if (fii != fiiEnd)
{
while (true)
{
if (pos == index.end() || *pos >= fii->start() + fii->numPoints())
{
count += std::distance(begin, pos);
appendCompressed(view, dims, *fii, begin, pos);
if (pos == index.end())
break;
begin = pos;
do
{
fii++;
if (fii == fiiEnd)
goto flush;
} while (*begin >= fii->start() + fii->numPoints());
}
pos++;
}
}
flush:
flushCompressed(table, view, o);
m_manager.logOctant(o.key(), count);
return pos;
}
void Processor::appendCompressed(pdal::PointViewPtr view, const DimInfoList& dims,
const FileInfo& fi, IndexIter begin, IndexIter end)
{
//ABELL - This could be improved by making a point table that handles a bunch
// of FileInfos/raw addresses. It would totally avoid the copy.
pdal::PointId pointId = view->size();
for (IndexIter it = begin; it != end; ++it)
{
char *base = fi.address() + ((*it - fi.start()) * m_b.pointSize);
for (const FileDimInfo& fdi : dims)
view->setField(fdi.dim, fdi.type, pointId,
reinterpret_cast<void *>(base + fdi.offset));
pointId++;
}
}
void Processor::flushCompressed(pdal::PointTableRef table, pdal::PointViewPtr view,
const OctantInfo& oi)
{
using namespace pdal;
std::string filename = m_b.outputDir + "/ept-data/" + oi.key().toString() + ".laz";
StageFactory factory;
BufferReader r;
r.addView(view);
Stage *prev = &r;
if (table.layout()->hasDim(Dimension::Id::GpsTime))
{
Stage *f = factory.createStage("filters.sort");
pdal::Options fopts;
fopts.add("dimension", "gpstime");
f->setOptions(fopts);
f->setInput(r);
prev = f;
}
Stage *w = factory.createStage("writers.las");
pdal::Options wopts;
wopts.add("extra_dims", "all");
wopts.add("software_id", "Entwine 1.0");
wopts.add("compression", "laszip");
wopts.add("filename", filename);
w->setOptions(wopts);
w->setInput(*prev);
// Set dataformat ID based on time/rgb, but for now accept the default.
w->prepare(table);
w->execute(table);
}
} // namespace bu
} // namespace untwine

65
external/untwine/bu/Processor.hpp vendored Normal file
View File

@ -0,0 +1,65 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <pdal/PointTable.hpp>
#include <pdal/PointView.hpp>
#include "BuTypes.hpp"
#include "PointAccessor.hpp"
#include "VoxelInfo.hpp"
namespace untwine
{
class GridKey;
namespace bu
{
class OctantInfo;
class PyramidManager;
class Processor
{
public:
Processor(PyramidManager& manager, const VoxelInfo&, const BaseInfo& b);
void run();
private:
using Index = std::deque<int>;
using IndexIter = Index::const_iterator;
void sample(Index& accepted, Index& rejected);
void write(Index& accepted, Index& rejected);
bool acceptable(int pointId, GridKey key);
bool tooClose(pdal::PointId id1, pdal::PointId id2);
void appendRemainder(Index& index);
void writeBinOutput(Index& index);
void writeCompressedOutput(Index& index);
IndexIter writeOctantCompressed(const OctantInfo& o, Index& index, IndexIter pos);
void appendCompressed(pdal::PointViewPtr view, const DimInfoList& dims, const FileInfo& fi,
IndexIter begin, IndexIter end);
void flushCompressed(pdal::PointTableRef table, pdal::PointViewPtr view,
const OctantInfo& oi);
VoxelInfo m_vi;
const BaseInfo& m_b;
PyramidManager& m_manager;
PointAccessor m_points;
};
} // namespace bu
} // namespace untwine

233
external/untwine/bu/PyramidManager.cpp vendored Normal file
View File

@ -0,0 +1,233 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <regex>
#include <string>
#include <vector>
#include <pdal/util/FileUtils.hpp>
#include "../untwine/ProgressWriter.hpp"
#include "../untwine/VoxelKey.hpp"
#include "Processor.hpp"
#include "PyramidManager.hpp"
#include "VoxelInfo.hpp"
namespace untwine
{
namespace bu
{
//PyramidManager::PyramidManager(const BaseInfo& b) : m_b(b), m_pool(6), m_totalPoints(0)
//PyramidManager::PyramidManager(const BaseInfo& b) : m_b(b), m_pool(8), m_totalPoints(0)
PyramidManager::PyramidManager(const BaseInfo& b) : m_b(b), m_pool(10), m_totalPoints(0)
{}
PyramidManager::~PyramidManager()
{}
void PyramidManager::setProgress(ProgressWriter *progress)
{
m_progress = progress;
}
void PyramidManager::queue(const OctantInfo& o)
{
{
std::lock_guard<std::mutex> lock(m_mutex);
m_queue.push(o);
}
m_cv.notify_one();
}
void PyramidManager::run()
{
while (true)
{
OctantInfo o;
{
std::unique_lock<std::mutex> lock(m_mutex);
m_cv.wait(lock, [this](){return m_queue.size();});
o = m_queue.front();
m_queue.pop();
}
if (o.key() == VoxelKey(0, 0, 0, 0))
break;
process(o);
}
createHierarchy();
}
// Take the item off the queue and stick it on the complete list. If we have all 8 octants,
// remove the items from the complete list and queue a Processor job.
void PyramidManager::process(const OctantInfo& o)
{
VoxelKey pk = o.key().parent();
addComplete(o);
if (!childrenComplete(pk))
return;
VoxelInfo vi(m_b.bounds, pk);
for (int i = 0; i < 8; ++i)
vi[i] = removeComplete(pk.child(i));
// If there are no points in this voxel, just queue it as a child.
if (!vi.hasPoints())
{
queue(vi.octant());
m_progress->writeIncrement("Bypass sample for " + vi.key().toString());
}
else
{
m_pool.add([vi, this]()
{
Processor p(*this, vi, m_b);
p.run();
m_progress->writeIncrement("Sample complete for " + vi.key().toString());
});
}
}
void PyramidManager::addComplete(const OctantInfo& o)
{
m_completes.insert({o.key(), o});
}
bool PyramidManager::childrenComplete(const VoxelKey& parent)
{
for (int i = 0; i < 8; ++i)
if (m_completes.find(parent.child(i)) == m_completes.end())
return false;
return true;
}
OctantInfo PyramidManager::removeComplete(const VoxelKey& k)
{
OctantInfo o;
auto oi = m_completes.find(k);
if (oi != m_completes.end())
{
o = std::move(oi->second);
m_completes.erase(oi);
}
return o;
}
void PyramidManager::logOctant(const VoxelKey& k, int cnt)
{
std::lock_guard<std::mutex> lock(m_mutex);
m_written.insert({k, cnt});
m_totalPoints += cnt;
}
void PyramidManager::createHierarchy()
{
std::function<int(const VoxelKey&)> calcCounts;
calcCounts = [this, &calcCounts](const VoxelKey& k)
{
int count = 0;
for (int i = 0; i < 8; ++i)
{
VoxelKey c = k.child(i);
if (m_written.find(c) != m_written.end())
count += calcCounts(c);
}
m_childCounts[k] = count;
return count + 1;
};
calcCounts(VoxelKey(0, 0, 0, 0));
std::deque<VoxelKey> roots;
roots.push_back(VoxelKey(0, 0, 0, 0));
while (roots.size())
{
VoxelKey k = roots.front();
roots.pop_front();
auto newRoots = emitRoot(k);
roots.insert(roots.end(), newRoots.begin(), newRoots.end());
}
}
std::deque<VoxelKey> PyramidManager::emitRoot(const VoxelKey& root)
{
int level = root.level();
int stopLevel = level + LevelBreak;
Entries entries;
entries.push_back({root, m_written[root]});
std::deque<VoxelKey> roots = emit(root, stopLevel, entries);
std::ofstream out(m_b.outputDir + "/ept-hierarchy/" + root.toString() + ".json");
out << "{\n";
for (auto it = entries.begin(); it != entries.end(); ++it)
{
if (it != entries.begin())
out << ",\n";
out << "\"" << it->first << "\": " << it->second;
}
out << "\n";
out << "}\n";
return roots;
}
std::deque<VoxelKey> PyramidManager::emit(const VoxelKey& p, int stopLevel, Entries& entries)
{
std::deque<VoxelKey> roots;
for (int i = 0; i < 8; ++i)
{
VoxelKey c = p.child(i);
auto ci = m_childCounts.find(c);
if (ci != m_childCounts.end())
{
if (c.level() != stopLevel || ci->second <= MinHierarchySize)
{
entries.push_back({c, m_written[c]});
auto r = emit(c, stopLevel, entries);
roots.insert(roots.end(), r.begin(), r.end());
}
else
{
entries.push_back({c, -1});
roots.push_back(c);
}
}
}
return roots;
}
} // namespace bu
} // namespace untwine

77
external/untwine/bu/PyramidManager.hpp vendored Normal file
View File

@ -0,0 +1,77 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <memory>
#include <mutex>
#include <unordered_map>
#include <vector>
#include <pdal/util/ThreadPool.hpp>
#include "BuTypes.hpp"
#include "OctantInfo.hpp"
namespace untwine
{
class ProgressWriter;
namespace bu
{
class OctantInfo;
class Processor;
class PyramidManager
{
using Entries = std::vector<std::pair<VoxelKey, int>>;
public:
PyramidManager(const BaseInfo& b);
~PyramidManager();
void setProgress(ProgressWriter *progress);
void queue(const OctantInfo& o);
void run();
void logOctant(const VoxelKey& k, int cnt);
uint64_t totalPoints() const
{ return m_totalPoints; }
private:
const int LevelBreak = 4;
const int MinHierarchySize = 50;
const BaseInfo& m_b;
std::mutex m_mutex;
std::condition_variable m_cv;
std::unordered_map<VoxelKey, OctantInfo> m_completes;
std::queue<OctantInfo> m_queue;
pdal::ThreadPool m_pool;
uint64_t m_totalPoints;
ProgressWriter *m_progress;
//
std::unordered_map<VoxelKey, int> m_written;
std::unordered_map<VoxelKey, int> m_childCounts;
void getInputFiles();
void process(const OctantInfo& o);
void addComplete(const OctantInfo& o);
bool childrenComplete(const VoxelKey& parent);
OctantInfo removeComplete(const VoxelKey& k);
//
void createHierarchy();
std::deque<VoxelKey> emitRoot(const VoxelKey& root);
std::deque<VoxelKey> emit(const VoxelKey& p, int stopLevel, Entries& entries);
};
} // namespace bu
} // namespace untwine

162
external/untwine/bu/VoxelInfo.hpp vendored Normal file
View File

@ -0,0 +1,162 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <array>
#include <unordered_map>
#include "../untwine/FileDimInfo.hpp"
#include "../untwine/GridKey.hpp"
#include "../untwine/Point.hpp"
#include "../untwine/VoxelKey.hpp"
#include "OctantInfo.hpp"
namespace untwine
{
namespace bu
{
class VoxelInfo
{
public:
// This probably needs a data structure geared for sparse data as I *think* that many
// of our lookups will fail.
using Grid = std::unordered_map<GridKey, int>;
VoxelInfo(const pdal::BOX3D& fullBounds, const VoxelKey& key) :
m_fullBounds(fullBounds), m_octant(key)
{
//ABELL - This shouldn't be necessary. The key should be in the children
// when they're pulled out of the queue.
for (int i = 0; i < 8; ++i)
m_children[i].setKey(key.child(i));
int cells = (int)std::pow(2, key.level());
m_xWidth = (fullBounds.maxx - fullBounds.minx) / cells;
m_yWidth = (fullBounds.maxy - fullBounds.miny) / cells;
m_zWidth = (fullBounds.maxz - fullBounds.minz) / cells;
// Calculate the bounds of this voxel.
m_bounds.minx = fullBounds.minx + (key.x() * m_xWidth);
m_bounds.maxx = m_bounds.minx + m_xWidth;
m_bounds.miny = fullBounds.miny + (key.y() * m_yWidth);
m_bounds.maxy = m_bounds.miny + m_yWidth;
m_bounds.minz = fullBounds.minz + (key.z() * m_zWidth);
m_bounds.maxz = m_bounds.minz + m_zWidth;
// Determine spacing between points.
// m_spacing = minWidth() / 128.0;
m_spacing = maxWidth() / 128.0;
// Make the spacing smaller than what we expect as the final spacing since we're
// going to select points from the grid for the parent.
if (key != VoxelKey(0, 0, 0, 0))
m_spacing *= 1.5;
m_squareSpacing = m_spacing * m_spacing;
static const double sqrt3 = std::sqrt(3);
m_gridCellWidth = m_spacing / sqrt3;
m_gridXCount = (int)std::ceil((m_bounds.maxx - m_bounds.minx) / m_gridCellWidth);
m_gridYCount = (int)std::ceil((m_bounds.maxy - m_bounds.miny) / m_gridCellWidth);
m_gridZCount = (int)std::ceil((m_bounds.maxz - m_bounds.minz) / m_gridCellWidth);
}
VoxelKey key() const
{ return m_octant.key(); }
OctantInfo& operator[](int dir)
{ return m_children[dir]; }
OctantInfo& octant()
{ return m_octant; }
size_t numPoints() const
{
size_t cnt = 0;
for (const OctantInfo& oi : m_children)
cnt += oi.numPoints();
return cnt;
}
bool hasPoints() const
{
for (const OctantInfo& oi : m_children)
if (oi.hasPoints())
return true;
return false;
}
double spacing() const
{ return m_spacing; }
double squareSpacing() const
{ return m_squareSpacing; }
double minWidth() const
{ return (std::min)((std::min)(m_xWidth, m_yWidth), m_zWidth); }
double maxWidth() const
{ return (std::max)((std::max)(m_xWidth, m_yWidth), m_zWidth); }
double xWidth() const
{ return m_xWidth; }
double yWidth() const
{ return m_yWidth; }
double zWidth() const
{ return m_zWidth; }
GridKey gridKey(const Point& p) const
{
double x = p.x() - m_bounds.minx;
double y = p.y() - m_bounds.miny;
double z = p.z() - m_bounds.minz;
return GridKey((int)(x / m_gridCellWidth), (int)(y / m_gridCellWidth), (int)(z / m_gridCellWidth));
}
//ABELL - Really torn WRT making Grid its own thing.
Grid& grid()
{ return m_grid; }
int gridXCount() const
{ return m_gridXCount; }
int gridYCount() const
{ return m_gridYCount; }
int gridZCount() const
{ return m_gridZCount; }
private:
pdal::BOX3D m_fullBounds;
pdal::BOX3D m_bounds;
double m_xWidth;
double m_yWidth;
double m_zWidth;
double m_gridCellWidth;
int m_gridXCount;
int m_gridYCount;
int m_gridZCount;
std::array<OctantInfo, 8> m_children;
OctantInfo m_octant;
double m_spacing;
double m_squareSpacing;
Grid m_grid;
};
} // namespace bu
} // namespace untwine

59
external/untwine/epf/BufferCache.cpp vendored Normal file
View File

@ -0,0 +1,59 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <mutex>
#include "BufferCache.hpp"
namespace untwine
{
namespace epf
{
// If we have a buffer in the cache, return it. Otherwise create a new one and return that.
DataVecPtr BufferCache::fetch()
{
std::unique_lock<std::mutex> lock(m_mutex);
m_cv.wait(lock, [this](){ return m_buffers.size() || m_count < MaxBuffers; });
if (m_buffers.size())
{
DataVecPtr buf(std::move(m_buffers.back()));
m_buffers.pop_back();
return buf;
}
// m_count tracks the number of created buffers. We only create MaxBuffers buffers.
// If we've created that many, we wait until one is available.
m_count++;
return DataVecPtr(new DataVec(BufSize));
}
// Put a buffer back in the cache.
void BufferCache::replace(DataVecPtr&& buf)
{
std::unique_lock<std::mutex> lock(m_mutex);
//ABELL - Fix this.
// buf->resize(BufSize);
m_buffers.push_back(std::move(buf));
if (m_count == MaxBuffers)
{
lock.unlock();
m_cv.notify_one();
}
}
} // namespace epf
} // namespace untwine

45
external/untwine/epf/BufferCache.hpp vendored Normal file
View File

@ -0,0 +1,45 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <deque>
#include <mutex>
#include <condition_variable>
#include "EpfTypes.hpp"
namespace untwine
{
namespace epf
{
// This is simply a cache of data buffers to avoid continuous allocation and deallocation.
class BufferCache
{
public:
BufferCache() : m_count(0)
{}
DataVecPtr fetch();
void replace(DataVecPtr&& buf);
private:
std::deque<DataVecPtr> m_buffers;
std::mutex m_mutex;
std::condition_variable m_cv;
int m_count;
};
} // namespace epf
} // namespace untwine

78
external/untwine/epf/Cell.cpp vendored Normal file
View File

@ -0,0 +1,78 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include "Cell.hpp"
#include "Writer.hpp"
namespace untwine
{
namespace epf
{
void Cell::initialize()
{
m_buf = m_writer->bufferCache().fetch();
m_pos = m_buf->data();
m_endPos = m_pos + m_pointSize * (BufSize / m_pointSize);
}
void Cell::write()
{
// Resize the buffer so the writer knows how much to write.
size_t size = m_pos - m_buf->data();
if (size)
// {
// m_buf->resize(size);
m_writer->enqueue(m_key, std::move(m_buf), size);
// }
}
void Cell::advance()
{
m_pos += m_pointSize;
if (m_pos >= m_endPos)
{
write();
initialize();
}
}
///
/// CellMgr
///
CellMgr::CellMgr(int pointSize, Writer *writer) : m_pointSize(pointSize), m_writer(writer)
{}
Cell *CellMgr::get(const VoxelKey& key)
{
auto it = m_cells.find(key);
if (it == m_cells.end())
{
std::unique_ptr<Cell> cell(new Cell(key, m_pointSize, m_writer));
it = m_cells.insert( {key, std::move(cell)} ).first;
}
Cell& c = *(it->second);
return &c;
}
void CellMgr::flush()
{
for (auto& cp : m_cells)
cp.second->write();
}
} // namespace epf
} // namespace untwine

79
external/untwine/epf/Cell.hpp vendored Normal file
View File

@ -0,0 +1,79 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <cstdint>
#include <cstddef>
#include <map>
#include <memory>
#include "EpfTypes.hpp"
#include "../untwine/Point.hpp"
#include "../untwine/VoxelKey.hpp"
namespace untwine
{
namespace epf
{
class Writer;
// A cell represents a voxel that contains points. All cells are the same size. A cell has
// a buffer which is filled by points. When the buffer is filled, it's passed to the writer
// and a new buffer is created.
class Cell
{
public:
Cell(const VoxelKey& key, int pointSize, Writer *writer) :
m_key(key), m_pointSize(pointSize), m_writer(writer)
{
assert(pointSize < BufSize);
initialize();
}
void initialize();
Point point()
{ return Point(m_pos); }
VoxelKey key() const
{ return m_key; }
void copyPoint(Point& b)
{ std::copy(b.data(), b.data() + m_pointSize, m_pos); }
void write();
void advance();
private:
DataVecPtr m_buf;
VoxelKey m_key;
int m_pointSize;
Writer *m_writer;
uint8_t *m_pos;
uint8_t *m_endPos;
};
class CellMgr
{
public:
CellMgr(int pointSize, Writer *writer);
Cell *get(const VoxelKey& key);
void flush();
private:
int m_pointSize;
Writer *m_writer;
std::map<VoxelKey, std::unique_ptr<Cell>> m_cells;
};
} // namespace epf
} // namespace untwine

261
external/untwine/epf/Epf.cpp vendored Normal file
View File

@ -0,0 +1,261 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include "Epf.hpp"
#include "EpfTypes.hpp"
#include "FileProcessor.hpp"
#include "Reprocessor.hpp"
#include "Writer.hpp"
#include "../untwine/Common.hpp"
#include <unordered_set>
#include <pdal/Dimension.hpp>
#include <pdal/PointLayout.hpp>
#include <pdal/StageFactory.hpp>
#include <pdal/util/Algorithm.hpp>
#include <pdal/util/Bounds.hpp>
#include <pdal/util/FileUtils.hpp>
#include <pdal/util/ProgramArgs.hpp>
namespace untwine
{
namespace epf
{
void writeMetadata(const std::string& tempDir, const Grid& grid,
const std::string& srs, const pdal::PointLayoutPtr& layout)
{
std::ofstream out(tempDir + "/" + MetadataFilename);
pdal::BOX3D b = grid.processingBounds();
std::ios init(NULL);
init.copyfmt(out);
out << std::setw(10) << std::fixed;
out << b.minx << " " << b.miny << " " << b.minz << "\n";
out << b.maxx << " " << b.maxy << " " << b.maxz << "\n";
out << "\n";
b = grid.conformingBounds();
out << b.minx << " " << b.miny << " " << b.minz << "\n";
out << b.maxx << " " << b.maxy << " " << b.maxz << "\n";
out << "\n";
out.copyfmt(init);
out << srs << "\n";
out << "\n";
for (pdal::Dimension::Id id : layout->dims())
out << layout->dimName(id) << " " << (int)layout->dimType(id) << " " <<
layout->dimOffset(id) << "\n";
}
/// Epf
Epf::Epf() : m_pool(8)
{}
Epf::~Epf()
{}
void Epf::run(const Options& options, ProgressWriter& progress)
{
using namespace pdal;
double millionPoints = 0;
BOX3D totalBounds;
if (pdal::FileUtils::fileExists(options.tempDir + "/" + MetadataFilename))
fatal("Output directory already contains EPT data.");
m_grid.setCubic(options.doCube);
std::vector<FileInfo> fileInfos;
progress.m_total = createFileInfo(options.inputFiles, options.dimNames, fileInfos);
if (options.level != -1)
m_grid.resetLevel(options.level);
// This is just a debug thing that will allow the number of input files to be limited.
if (fileInfos.size() > m_fileLimit)
fileInfos.resize(m_fileLimit);
// Stick all the dimension names from each input file in a set.
std::unordered_set<std::string> allDimNames;
for (const FileInfo& fi : fileInfos)
for (const FileDimInfo& fdi : fi.dimInfo)
allDimNames.insert(fdi.name);
// Register the dimensions, either as the default type or double if we don't know
// what it is.
PointLayoutPtr layout(new PointLayout());
for (const std::string& dimName : allDimNames)
{
Dimension::Type type = Dimension::defaultType(Dimension::id(dimName));
if (type == Dimension::Type::None)
type = Dimension::Type::Double;
layout->registerOrAssignDim(dimName, type);
}
layout->finalize();
// Fill in dim info now that the layout is finalized.
for (FileInfo& fi : fileInfos)
{
for (FileDimInfo& di : fi.dimInfo)
{
di.dim = layout->findDim(di.name);
di.type = layout->dimType(di.dim);
di.offset = layout->dimOffset(di.dim);
}
}
// Make a writer with 4 threads.
m_writer.reset(new Writer(options.tempDir, 4, layout->pointSize()));
// Sort file infos so the largest files come first. This helps to make sure we don't delay
// processing big files that take the longest (use threads more efficiently).
std::sort(fileInfos.begin(), fileInfos.end(), [](const FileInfo& f1, const FileInfo& f2)
{ return f1.numPoints > f2.numPoints; });
progress.m_threshold = progress.m_total / 40;
progress.setIncrement(.01);
progress.m_current = 0;
// Add the files to the processing pool
for (const FileInfo& fi : fileInfos)
{
int pointSize = layout->pointSize();
m_pool.add([&fi, &progress, pointSize, this]()
{
FileProcessor fp(fi, pointSize, m_grid, m_writer.get(), progress);
fp.run();
});
}
// Wait for all the processors to finish and restart.
m_pool.cycle();
progress.setPercent(.4);
// Tell the writer that it can exit. stop() will block until the writer threads
// are finished.
m_writer->stop();
// Get totals from the current writer.
//ABELL - would be nice to avoid this copy, but it probably doesn't matter much.
Totals totals = m_writer->totals(MaxPointsPerNode);
progress.setPercent(.4);
progress.setIncrement(.2 / totals.size());
// Make a new writer.
m_writer.reset(new Writer(options.tempDir, 4, layout->pointSize()));
for (auto& t : totals)
{
VoxelKey key = t.first;
int numPoints = t.second;
int pointSize = layout->pointSize();
std::string tempDir = options.tempDir;
m_pool.add([&progress, key, numPoints, pointSize, tempDir, this]()
{
Reprocessor r(key, numPoints, pointSize, tempDir, m_grid, m_writer.get());
r.run();
progress.writeIncrement("Reprocessed voxel " + key.toString());
});
}
m_pool.stop();
m_writer->stop();
std::string srs = m_srsFileInfo.valid() ? m_srsFileInfo.srs.getWKT() : "NONE";
writeMetadata(options.tempDir, m_grid, srs, layout);
}
PointCount Epf::createFileInfo(const StringList& input, StringList dimNames,
std::vector<FileInfo>& fileInfos)
{
using namespace pdal;
std::vector<std::string> filenames;
PointCount totalPoints = 0;
// If there are some dim names specified, make sure they contain X, Y and Z and that
// they're all uppercase.
if (!dimNames.empty())
{
for (std::string& d : dimNames)
d = Utils::toupper(d);
for (const std::string& xyz : { "X", "Y", "Z" })
if (!Utils::contains(dimNames, xyz))
dimNames.push_back(xyz);
}
// If any of the specified input files is a directory, get the names of the files
// in the directory and add them.
for (const std::string& filename : input)
{
if (FileUtils::isDirectory(filename))
{
std::vector<std::string> dirfiles = FileUtils::directoryList(filename);
filenames.insert(filenames.end(), dirfiles.begin(), dirfiles.end());
}
else
filenames.push_back(filename);
}
// Determine a driver for each file and get a preview of the file. If we couldn't
// Create a FileInfo object containing the file bounds, dimensions, filename and
// associated driver. Expand our grid by the bounds and file point count.
for (std::string& filename : filenames)
{
StageFactory factory;
std::string driver = factory.inferReaderDriver(filename);
if (driver.empty())
fatal("Can't infer reader for '" + filename + "'.");
Stage *s = factory.createStage(driver);
pdal::Options opts;
opts.add("filename", filename);
s->setOptions(opts);
QuickInfo qi = s->preview();
if (!qi.valid())
throw "Couldn't get quick info for '" + filename + "'.";
FileInfo fi;
fi.bounds = qi.m_bounds;
fi.numPoints = qi.m_pointCount;
fi.filename = filename;
fi.driver = driver;
// Accept dimension names if there are no limits or this name is in the list
// of desired dimensions.
for (const std::string& name : qi.m_dimNames)
if (dimNames.empty() || Utils::contains(dimNames, Utils::toupper(name)))
fi.dimInfo.push_back(FileDimInfo(name));
if (m_srsFileInfo.valid() && m_srsFileInfo.srs != qi.m_srs)
std::cerr << "Files have mismatched SRS values. Using SRS from '" <<
m_srsFileInfo.filename << "'.\n";
fi.srs = qi.m_srs;
fileInfos.push_back(fi);
if (!m_srsFileInfo.valid() && qi.m_srs.valid())
m_srsFileInfo = fi;
m_grid.expand(qi.m_bounds, qi.m_pointCount);
totalPoints += fi.numPoints;
}
return totalPoints;
}
} // namespace epf
} // namespace untwine

58
external/untwine/epf/Epf.hpp vendored Normal file
View File

@ -0,0 +1,58 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <map>
#include <vector>
#include <pdal/SpatialReference.hpp>
#include <pdal/util/ThreadPool.hpp>
#include "EpfTypes.hpp"
#include "Grid.hpp"
#include "../untwine/ProgressWriter.hpp"
namespace untwine
{
struct Options;
class ProgressWriter;
namespace epf
{
struct FileInfo;
class Writer;
class Epf
{
public:
Epf();
~Epf();
void run(const Options& options, ProgressWriter& progress);
private:
PointCount createFileInfo(const StringList& input, StringList dimNames,
std::vector<FileInfo>& fileInfos);
Grid m_grid;
std::unique_ptr<Writer> m_writer;
pdal::ThreadPool m_pool;
size_t m_fileLimit;
FileInfo m_srsFileInfo;
};
} // namespace epf
} // namespace untwine

54
external/untwine/epf/EpfTypes.hpp vendored Normal file
View File

@ -0,0 +1,54 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <pdal/Dimension.hpp>
#include <pdal/SpatialReference.hpp>
#include <pdal/util/Bounds.hpp>
#include <cstdint>
#include <memory>
#include <unordered_map>
#include <vector>
#include "../untwine/FileDimInfo.hpp"
#include "../untwine/VoxelKey.hpp"
namespace untwine
{
namespace epf
{
using DataVec = std::vector<uint8_t>;
using DataVecPtr = std::unique_ptr<DataVec>;
using Totals = std::unordered_map<VoxelKey, size_t>;
constexpr int MaxPointsPerNode = 100000;
constexpr int BufSize = 4096 * 10;
constexpr int MaxBuffers = 1000;
struct FileInfo
{
std::string filename;
std::string driver;
DimInfoList dimInfo;
uint64_t numPoints;
pdal::BOX3D bounds;
pdal::SpatialReference srs;
bool valid() const
{ return filename.size(); }
};
} // namespace epf
} // namespace untwine

97
external/untwine/epf/FileProcessor.cpp vendored Normal file
View File

@ -0,0 +1,97 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include "FileProcessor.hpp"
#include "../untwine/ProgressWriter.hpp"
#include <pdal/StageFactory.hpp>
#include <pdal/filters/StreamCallbackFilter.hpp>
namespace untwine
{
namespace epf
{
FileProcessor::FileProcessor(const FileInfo& fi, size_t pointSize, const Grid& grid,
Writer *writer, ProgressWriter& progress) :
m_fi(fi), m_cellMgr(pointSize, writer), m_grid(grid), m_progress(progress)
{}
void FileProcessor::run()
{
pdal::Options opts;
opts.add("filename", m_fi.filename);
pdal::StageFactory factory;
pdal::Stage *s = factory.createStage(m_fi.driver);
s->setOptions(opts);
pdal::StreamCallbackFilter f;
const PointCount CountIncrement = 100000;
PointCount count = 0;
PointCount limit = CountIncrement;
// We need to move the data from the PointRef to some output buffer. We copy the data
// to the end of the *last* output buffer we used in hopes that it's the right one.
// If it's not we lose and we're forced to move that data to the another cell,
// which then becomes the active cell.
// This is some random cell that ultimately won't get used, but it contains a buffer
// into which we can write data.
Cell *cell = m_cellMgr.get(VoxelKey());
f.setCallback([this, &count, &limit, &cell](pdal::PointRef& point)
{
// Write the data into the point buffer in the cell. This is the *last*
// cell buffer that we used. We're hoping that it's the right one.
Point p = cell->point();
for (const FileDimInfo& fdi : m_fi.dimInfo)
point.getField(reinterpret_cast<char *>(p.data() + fdi.offset),
fdi.dim, fdi.type);
// Find the actual cell that this point belongs in. If it's not the one
// we chose, copy the data to the correct cell.
VoxelKey cellIndex = m_grid.key(p.x(), p.y(), p.z());
if (cellIndex != cell->key())
{
cell = m_cellMgr.get(cellIndex);
cell->copyPoint(p);
}
// Advance the cell - move the buffer pointer so when refer to the cell's
// point, we're referring to the next location in the cell's buffer.
cell->advance();
count++;
if (count == limit)
{
m_progress.update(CountIncrement);
limit += CountIncrement;
}
return true;
}
);
f.setInput(*s);
pdal::FixedPointTable t(1000);
f.prepare(t);
f.execute(t);
m_progress.update(count % CountIncrement);
// Flush any data remaining in the cells.
m_cellMgr.flush();
}
} // namespace epf
} // namespace untwine

46
external/untwine/epf/FileProcessor.hpp vendored Normal file
View File

@ -0,0 +1,46 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include "EpfTypes.hpp"
#include "Grid.hpp"
#include "Cell.hpp"
namespace untwine
{
class ProgressWriter;
namespace epf
{
class Writer;
// Processes a single input file (FileInfo) and writes data to the Writer.
class FileProcessor
{
public:
FileProcessor(const FileInfo& fi, size_t pointSize, const Grid& grid, Writer *writer,
ProgressWriter& progress);
Cell *getCell(const VoxelKey& key);
void run();
private:
FileInfo m_fi;
CellMgr m_cellMgr;
Grid m_grid;
ProgressWriter& m_progress;
};
} // namespace epf
} // namespace untwine

106
external/untwine/epf/Grid.cpp vendored Normal file
View File

@ -0,0 +1,106 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <cmath>
#include <cstdint>
#include "Epf.hpp"
#include "Grid.hpp"
using namespace pdal;
namespace untwine
{
namespace epf
{
void Grid::expand(const BOX3D& bounds, size_t points)
{
m_bounds.grow(bounds);
double xside = m_bounds.maxx - m_bounds.minx;
double yside = m_bounds.maxy - m_bounds.miny;
double zside = m_bounds.maxz - m_bounds.minz;
double side = (std::max)(xside, (std::max)(yside, zside));
m_cubicBounds = BOX3D(m_bounds.minx, m_bounds.miny, m_bounds.minz,
m_bounds.minx + side, m_bounds.miny + side, m_bounds.minz + side);
m_millionPoints += size_t(points / 1000000.0);
resetLevel(calcLevel());
}
int Grid::calcLevel()
{
int level = 0;
double mp = (double)m_millionPoints;
double limit = (MaxPointsPerNode / 1000000.0);
double xside = m_bounds.maxx - m_bounds.minx;
double yside = m_bounds.maxy - m_bounds.miny;
double zside = m_bounds.maxz - m_bounds.minz;
double side = (std::max)(xside, (std::max)(yside, zside));
while (mp > MaxPointsPerNode / 1000000.0)
{
if (m_cubic)
{
if (xside >= side)
mp /= 2;
if (yside >= side)
mp /= 2;
if (zside >= side)
mp /= 2;
}
else
mp /= 8;
side /= 2;
level++;
}
return level;
}
void Grid::resetLevel(int level)
{
// We have to have at least level 1 or things break when sampling.
m_maxLevel = (std::max)(level, 1);
m_gridSize = (int)std::pow(2, level);
if (m_cubic)
{
m_xsize = (m_cubicBounds.maxx - m_cubicBounds.minx) / m_gridSize;
m_ysize = m_xsize;
m_zsize = m_xsize;
}
else
{
m_xsize = (m_bounds.maxx - m_bounds.minx) / m_gridSize;
m_ysize = (m_bounds.maxy - m_bounds.miny) / m_gridSize;
m_zsize = (m_bounds.maxz - m_bounds.minz) / m_gridSize;
}
}
VoxelKey Grid::key(double x, double y, double z)
{
int xi = (int)std::floor((x - m_bounds.minx) / m_xsize);
int yi = (int)std::floor((y - m_bounds.miny) / m_ysize);
int zi = (int)std::floor((z - m_bounds.minz) / m_zsize);
xi = (std::min)((std::max)(0, xi), m_gridSize - 1);
yi = (std::min)((std::max)(0, yi), m_gridSize - 1);
zi = (std::min)((std::max)(0, zi), m_gridSize - 1);
return VoxelKey(xi, yi, zi, m_maxLevel);
}
} // namespace epf
} // namespace untwine

60
external/untwine/epf/Grid.hpp vendored Normal file
View File

@ -0,0 +1,60 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <pdal/util/Bounds.hpp>
#include "../untwine/VoxelKey.hpp"
namespace untwine
{
namespace epf
{
class Grid
{
public:
Grid() : m_gridSize(-1), m_maxLevel(-1), m_millionPoints(0), m_cubic(true)
{}
void expand(const pdal::BOX3D& bounds, size_t points);
int calcLevel();
void resetLevel(int level);
VoxelKey key(double x, double y, double z);
pdal::BOX3D processingBounds() const
{ return m_cubic ? m_cubicBounds : m_bounds; }
pdal::BOX3D cubicBounds() const
{ return m_cubicBounds; }
pdal::BOX3D conformingBounds() const
{ return m_bounds; }
int maxLevel() const
{ return m_maxLevel; }
void setCubic(bool cubic)
{ m_cubic = cubic; }
private:
int m_gridSize;
int m_maxLevel;
pdal::BOX3D m_bounds;
pdal::BOX3D m_cubicBounds;
size_t m_millionPoints;
bool m_cubic;
double m_xsize;
double m_ysize;
double m_zsize;
};
} // namespace epf
} // namespace untwine

75
external/untwine/epf/Reprocessor.cpp vendored Normal file
View File

@ -0,0 +1,75 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <pdal/util/FileUtils.hpp>
#include "Reprocessor.hpp"
#include "../untwine/Common.hpp"
namespace untwine
{
namespace epf
{
Reprocessor::Reprocessor(const VoxelKey& k, int numPoints, int pointSize,
const std::string& outputDir, Grid grid, Writer *writer) :
m_pointSize(pointSize), m_numPoints(numPoints), m_fileSize(pointSize * (size_t)numPoints),
m_grid(grid), m_mgr(pointSize, writer)
{
// We make an assumption that at most twice the number of points will be in a cell
// than there would be if the distribution was uniform, so we calculate based on
// each level breaking the points into 4.
// So, to find the number of levels, we need to solve for n:
//
// numPoints / (4^n) = MaxPointsPerNode
// =>
// numPoints / MaxPointsPerNode = 2^(2n)
// =>
// log2(numPoints / MaxPointsPerNode) = 2n
m_levels = (int)std::ceil(log2((double)numPoints / MaxPointsPerNode) / 2);
// We're going to steal points from the leaf nodes for sampling, so unless the
// spatial distribution is really off, this should be fine and pretty conservative.
m_grid.resetLevel(m_grid.maxLevel() + m_levels);
m_filename = outputDir + "/" + k.toString() + ".bin";
}
void Reprocessor::run()
{
auto ctx = pdal::FileUtils::mapFile(m_filename, true, 0, m_fileSize);
if (ctx.addr() == nullptr)
{
std::cerr << "FATAL: " + m_filename + ": " + ctx.what();
exit(-1);
}
// Wow, this is simple. How nice. The writer should get invoked automatically.
uint8_t *pos = reinterpret_cast<uint8_t *>(ctx.addr());
for (size_t i = 0; i < m_numPoints; ++i)
{
Point p(pos);
VoxelKey k = m_grid.key(p.x(), p.y(), p.z());
Cell *cell = m_mgr.get(k);
cell->copyPoint(p);
cell->advance();
pos += m_pointSize;
}
m_mgr.flush();
pdal::FileUtils::unmapFile(ctx);
pdal::FileUtils::deleteFile(m_filename);
}
} // namespace epf
} // namespace untwine

46
external/untwine/epf/Reprocessor.hpp vendored Normal file
View File

@ -0,0 +1,46 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include "EpfTypes.hpp"
#include "Grid.hpp"
#include "Cell.hpp"
namespace untwine
{
namespace epf
{
class Writer;
class Reprocessor
{
public:
Reprocessor(const VoxelKey& k, int numPoints, int pointSize, const std::string& outputDir,
Grid grid, Writer *writer);
void run();
private:
int m_pointSize;
uint64_t m_numPoints;
uint64_t m_fileSize;
Grid m_grid;
int m_levels;
CellMgr m_mgr;
std::string m_filename;
};
} // namespace epf
} // namespace untwine

134
external/untwine/epf/Writer.cpp vendored Normal file
View File

@ -0,0 +1,134 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <vector>
#include <pdal/util/FileUtils.hpp>
#include "Writer.hpp"
#include "Epf.hpp"
#include "../untwine/Common.hpp"
#include "../untwine/VoxelKey.hpp"
using namespace pdal;
namespace untwine
{
namespace epf
{
Writer::Writer(const std::string& directory, int numThreads, size_t pointSize) :
m_directory(directory), m_pool(numThreads), m_stop(false), m_pointSize(pointSize)
{
if (FileUtils::fileExists(directory))
{
if (!FileUtils::isDirectory(directory))
fatal("Specified output directory '" + directory + "' is not a directory.");
}
else
FileUtils::createDirectory(directory);
std::function<void()> f = std::bind(&Writer::run, this);
while (numThreads--)
m_pool.add(f);
}
std::string Writer::path(const VoxelKey& key)
{
return m_directory + "/" + key.toString() + ".bin";
}
Totals Writer::totals(size_t minSize)
{
Totals t;
for (auto ti = m_totals.begin(); ti != m_totals.end(); ++ti)
if (ti->second >= minSize)
t.insert(*ti);
return t;
}
void Writer::enqueue(const VoxelKey& key, DataVecPtr data, size_t dataSize)
{
{
std::lock_guard<std::mutex> lock(m_mutex);
m_totals[key] += (dataSize / m_pointSize);
m_queue.push_back({key, std::move(data), dataSize});
}
m_available.notify_one();
}
void Writer::stop()
{
{
std::lock_guard<std::mutex> lock(m_mutex);
m_stop = true;
}
m_available.notify_all();
m_pool.join();
}
void Writer::run()
{
while (true)
{
WriteData wd;
// Loop waiting for data.
while (true)
{
std::unique_lock<std::mutex> lock(m_mutex);
// Look for a queue entry that represents a key that we aren't already
// actively processing.
//ABELL - Perhaps a writer should grab and write *all* the entries in the queue
// that match the key we found.
auto li = m_queue.begin();
for (; li != m_queue.end(); ++li)
if (std::find(m_active.begin(), m_active.end(), li->key) == m_active.end())
break;
// If there is no data to process, exit if we're stopping. Wait otherwise.
// If there is data to process, stick the key on the active list and
// remove the item from the queue and break to do the actual write.
if (li == m_queue.end())
{
if (m_stop)
return;
m_available.wait(lock);
}
else
{
m_active.push_back(li->key);
wd = std::move(*li);
m_queue.erase(li);
break;
}
}
// Open the file. Write the data. Stick the buffer back on the cache.
// Remove the key from the active key list.
std::ofstream out(path(wd.key), std::ios::app | std::ios::binary);
out.write(reinterpret_cast<const char *>(wd.data->data()), wd.dataSize);
out.close();
if (!out)
fatal("Failure writing to '" + path(wd.key) + "'.");
m_bufferCache.replace(std::move(wd.data));
std::lock_guard<std::mutex> lock(m_mutex);
m_active.remove(wd.key);
}
}
} // namespace epf
} // namespace untwine

69
external/untwine/epf/Writer.hpp vendored Normal file
View File

@ -0,0 +1,69 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <condition_variable>
#include <list>
#include <mutex>
#include <string>
#include <pdal/util/ThreadPool.hpp>
#include "EpfTypes.hpp"
#include "BufferCache.hpp"
#include "../untwine/VoxelKey.hpp"
namespace untwine
{
namespace epf
{
class Writer
{
struct WriteData
{
VoxelKey key;
DataVecPtr data;
size_t dataSize;
};
public:
Writer(const std::string& directory, int numThreads, size_t pointSize);
void enqueue(const VoxelKey& key, DataVecPtr data, size_t dataSize);
void stop();
BufferCache& bufferCache()
{ return m_bufferCache; }
const Totals& totals()
{ return m_totals; }
Totals totals(size_t minSize);
private:
std::string path(const VoxelKey& key);
void run();
std::string m_directory;
pdal::ThreadPool m_pool;
BufferCache m_bufferCache;
bool m_stop;
size_t m_pointSize;
std::list<WriteData> m_queue;
std::list<VoxelKey> m_active;
Totals m_totals;
std::mutex m_mutex;
std::condition_variable m_available;
};
} // namespace epf
} // namespace untwine

29
external/untwine/untwine/Common.hpp vendored Normal file
View File

@ -0,0 +1,29 @@
#pragma once
#include <stdint.h>
#include <string>
#include <vector>
namespace untwine
{
using PointCount = uint64_t;
using StringList = std::vector<std::string>;
void fatal(const std::string& err);
struct Options
{
std::string outputDir;
StringList inputFiles;
std::string tempDir;
bool doCube;
size_t fileLimit;
int level;
int progressFd;
StringList dimNames;
};
const std::string MetadataFilename {"info2.txt"};
} // namespace untwine

6
external/untwine/untwine/Config.hpp.in vendored Normal file
View File

@ -0,0 +1,6 @@
#pragma once
#define UNTWINE_VERSION "@PROJECT_VERSION@"
#define UNTWINE_VERSION_MAJOR "@PROJECT_VERSION_MAJOR@"
#define UNTWINE_VERSION_MINOR "@PROJECT_VERSION_MINOR@"
#define UNTWINE_VERSION_PATCH "@PROJECT_VERSION_PATCH@"

47
external/untwine/untwine/FileDimInfo.hpp vendored Normal file
View File

@ -0,0 +1,47 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <pdal/Dimension.hpp>
namespace untwine
{
struct FileDimInfo
{
FileDimInfo()
{}
FileDimInfo(const std::string& name) : name(name)
{}
std::string name;
pdal::Dimension::Type type;
int offset;
pdal::Dimension::Id dim;
};
using DimInfoList = std::vector<FileDimInfo>;
inline std::ostream& operator<<(std::ostream& out, const FileDimInfo& fdi)
{
out << fdi.name << " " << (int)fdi.type << " " << fdi.offset;
return out;
}
inline std::istream& operator>>(std::istream& in, FileDimInfo& fdi)
{
in >> fdi.name >> (int&)fdi.type >> fdi.offset;
return in;
}
} // namespace untwine

71
external/untwine/untwine/GridKey.hpp vendored Normal file
View File

@ -0,0 +1,71 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
#include <cassert>
#include <climits>
#include <iostream>
namespace untwine
{
class GridKey
{
public:
GridKey(int i, int j, int k)
{
assert(i < (std::numeric_limits<uint8_t>::max)());
assert(j < (std::numeric_limits<uint8_t>::max)());
assert(k < (std::numeric_limits<uint8_t>::max)());
m_key = (i << (2 * CHAR_BIT)) | (j << CHAR_BIT) | k;
}
int i() const
{ return (m_key >> (2 * CHAR_BIT)); }
int j() const
{ return ((m_key >> (CHAR_BIT)) & 0xFF); }
int k() const
{ return (m_key & 0xFF); }
int key() const
{ return m_key; }
private:
int m_key;
};
inline bool operator==(const GridKey& k1, const GridKey& k2)
{
return k1.key() == k2.key();
}
inline std::ostream& operator<<(std::ostream& out, const GridKey& k)
{
out << k.i() << "/" << k.j() << "/" << k.k();
return out;
}
} // namespace untwine
namespace std
{
template<> struct hash<untwine::GridKey>
{
std::size_t operator()(const untwine::GridKey& k) const noexcept
{
return std::hash<int>()(k.key());
}
};
}

61
external/untwine/untwine/Point.hpp vendored Normal file
View File

@ -0,0 +1,61 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
namespace untwine
{
// Utterly trivial wrapper around a pointer.
class Point
{
public:
Point() : m_data(nullptr)
{}
Point(uint8_t *data) : m_data(data)
{}
Point(char *data) : m_data(reinterpret_cast<uint8_t *>(data))
{}
uint8_t *data()
{ return m_data; }
double x() const
{
double d;
memcpy(&d, ddata(), sizeof(d));
return d;
}
double y() const
{
double d;
memcpy(&d, ddata() + 1, sizeof(d));
return d;
}
double z() const
{
double d;
memcpy(&d, ddata() + 2, sizeof(d));
return d;
}
char *cdata() const
{ return reinterpret_cast<char *>(m_data); }
double *ddata() const
{ return reinterpret_cast<double *>(m_data); }
private:
uint8_t *m_data;
};
} // namespace untwine

93
external/untwine/untwine/ProgressWriter.cpp vendored Normal file
View File

@ -0,0 +1,93 @@
#include <cmath>
#include <string>
#include <mutex>
#ifndef _WIN32
#include <unistd.h>
#else
#include <Windows.h>
#endif
#include "ProgressWriter.hpp"
namespace untwine
{
ProgressWriter::ProgressWriter(int fd) : m_progressFd(fd), m_percent(0.0), m_increment(.1)
{}
void ProgressWriter::setIncrement(double increment)
{
if (!m_progressFd)
return;
std::unique_lock<std::mutex> lock(m_mutex);
m_increment = increment;
}
void ProgressWriter::setPercent(double percent)
{
if (!m_progressFd)
return;
std::unique_lock<std::mutex> lock(m_mutex);
m_percent = (std::max)(0.0, ((std::min)(1.0, percent)));
}
void ProgressWriter::writeIncrement(const std::string& message)
{
if (!m_progressFd)
return;
std::unique_lock<std::mutex> lock(m_mutex);
m_percent += m_increment;
m_percent = (std::min)(1.0, m_percent);
uint32_t percent = (uint32_t)std::round(m_percent * 100.0);
writeMessage(percent, message);
}
void ProgressWriter::write(double percent, const std::string& message)
{
if (!m_progressFd)
return;
std::unique_lock<std::mutex> lock(m_mutex);
m_percent = (std::min)(0.0, ((std::max)(1.0, percent)));
uint32_t ipercent = (uint32_t)std::round(m_percent * 100.0);
writeMessage(ipercent, message);
}
void ProgressWriter::writeMessage(uint32_t percent, const std::string& message)
{
#ifndef _WIN32
::write(m_progressFd, &percent, sizeof(percent));
uint32_t ssize = (uint32_t)message.size();
::write(m_progressFd, &ssize, sizeof(ssize));
::write(m_progressFd, message.data(), ssize);
#else
DWORD numWritten;
HANDLE h = reinterpret_cast<HANDLE>((intptr_t)m_progressFd);
WriteFile(h, &percent, sizeof(percent), &numWritten, NULL);
uint32_t ssize = (uint32_t)message.size();
WriteFile(h, &ssize, sizeof(ssize), &numWritten, NULL);
WriteFile(h, message.data(), ssize, &numWritten, NULL);
#endif
}
void ProgressWriter::update(PointCount count)
{
std::unique_lock<std::mutex> lock(m_mutex);
PointCount inc = m_current / m_threshold;
m_current += count;
PointCount postInc = m_current / m_threshold;
if (inc != postInc)
{
lock.unlock();
writeIncrement("Processed " + std::to_string(m_current) + " points");
}
}
} // namespace untwine

40
external/untwine/untwine/ProgressWriter.hpp vendored Normal file
View File

@ -0,0 +1,40 @@
#pragma once
#include <mutex>
#include "../untwine/Common.hpp"
namespace untwine
{
class ProgressWriter
{
public:
ProgressWriter(int fd);
/// Set the increment to use on the next call to setIncrement.
void setIncrement(double increment);
/// Set the absolute percentage to use for the next setIncrement.
void setPercent(double percent);
/// Write a message using the current increment.
void writeIncrement(const std::string& message);
/// Write a message and set the current percentage.
void write(double percent, const std::string& message);
void update(PointCount numProcessed);
// Utility fields
PointCount m_total;
PointCount m_threshold;
PointCount m_current;
private:
std::mutex m_mutex;
int m_progressFd;
double m_percent; // Current percent.
double m_increment; // Current increment.
void writeMessage(uint32_t percent, const std::string& message);
};
} //namespace untwine

125
external/untwine/untwine/Untwine.cpp vendored Normal file
View File

@ -0,0 +1,125 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#include <pdal/pdal_types.hpp>
#include <pdal/util/ProgramArgs.hpp>
#include "Common.hpp"
#include "Config.hpp"
#include "ProgressWriter.hpp"
#include "../epf/Epf.hpp"
#include "../bu/BuPyramid.hpp"
namespace untwine
{
void fatal(const std::string& err)
{
std::cerr << "untwine fatal error: " << err << "\n";
exit(-1);
}
void addArgs(pdal::ProgramArgs& programArgs, Options& options, pdal::Arg * &tempArg)
{
programArgs.add("files,i", "Input files/directory", options.inputFiles).setPositional();
programArgs.add("output_dir,o", "Output directory", options.outputDir).setPositional();
tempArg = &(programArgs.add("temp_dir", "Temp directory", options.tempDir));
programArgs.add("cube", "Make a cube, rather than a rectangular solid", options.doCube, true);
programArgs.add("level", "Set an initial tree leve, rather than guess based on data",
options.level, -1);
programArgs.add("file_limit", "Only load 'file_limit' files, even if more exist",
options.fileLimit, (size_t)10000000);
programArgs.add("progress_fd", "File descriptor on which to write process messages.",
options.progressFd);
programArgs.add("dims", "Dimensions to load. Note that X, Y and Z are always "
"loaded.", options.dimNames);
}
bool handleOptions(pdal::StringList& arglist, Options& options)
{
pdal::ProgramArgs programArgs;
pdal::Arg *tempArg;
addArgs(programArgs, options, tempArg);
try
{
bool version;
bool help;
programArgs.add("version", "Report the untwine version.", version);
programArgs.add("help", "Print some help.", help);
programArgs.parseSimple(arglist);
if (version)
std::cout << "untwine version (" << UNTWINE_VERSION << ")\n";
if (help)
{
std::cout << "Usage: untwine <options>\n";
programArgs.dump(std::cout, 2, 80);
}
if (help || version)
return false;
programArgs.parse(arglist);
if (!tempArg->set())
options.tempDir = options.outputDir + "/temp";
}
catch (const pdal::arg_error& err)
{
fatal(err.what());
}
return true;
}
void createDirs(const Options& options)
{
pdal::FileUtils::createDirectory(options.outputDir);
pdal::FileUtils::createDirectory(options.tempDir);
pdal::FileUtils::deleteFile(options.outputDir + "/ept.json");
pdal::FileUtils::deleteDirectory(options.outputDir + "/ept-data");
pdal::FileUtils::deleteDirectory(options.outputDir + "/ept-hierarchy");
pdal::FileUtils::createDirectory(options.outputDir + "/ept-data");
pdal::FileUtils::createDirectory(options.outputDir + "/ept-hierarchy");
}
} // namespace untwine
int main(int argc, char *argv[])
{
std::vector<std::string> arglist;
// Skip the program name.
argv++;
argc--;
while (argc--)
arglist.push_back(*argv++);
using namespace untwine;
Options options;
if (!handleOptions(arglist, options))
return 0;
createDirs(options);
ProgressWriter progress(options.progressFd);
epf::Epf preflight;
preflight.run(options, progress);
bu::BuPyramid builder;
builder.run(options, progress);
return 0;
}

117
external/untwine/untwine/VoxelKey.hpp vendored Normal file
View File

@ -0,0 +1,117 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#pragma once
namespace untwine
{
// This key supports large levels, but requires a larger key.
class VoxelKey
{
public:
VoxelKey() : m_x(0), m_y(0), m_z(0), m_level(0)
{}
// Key without level.
VoxelKey(int x, int y, int z) : m_x(x), m_y(y), m_z(z), m_level(0)
{}
VoxelKey(int x, int y, int z, int level) : m_x(x), m_y(y), m_z(z), m_level(level)
{}
VoxelKey child(int dir) const
{
return VoxelKey(
(m_x << 1) | (dir & 0x1),
(m_y << 1) | ((dir >> 1) & 0x1),
(m_z << 1) | ((dir >> 2) & 0x1),
m_level + 1);
}
VoxelKey parent() const
{
return VoxelKey(m_x >> 1, m_y >> 1, m_z >> 1, (std::max)(m_level - 1, 0));
}
int x() const
{ return m_x; }
int y() const
{ return m_y; }
int z() const
{ return m_z; }
int level() const
{ return m_level; }
std::string toString() const
{ return (std::string)(*this); }
operator std::string() const
{
return std::to_string(m_level) + '-' + std::to_string(m_x) + '-' +
std::to_string(m_y) + '-' + std::to_string(m_z);
}
private:
int m_x;
int m_y;
int m_z;
int m_level;
};
inline bool operator==(const VoxelKey& k1, const VoxelKey& k2)
{
return k1.x() == k2.x() && k1.y() == k2.y() && k1.z() == k2.z() && k1.level() == k2.level();
}
inline bool operator!=(const VoxelKey& k1, const VoxelKey& k2)
{
return k1.x() != k2.x() || k1.y() != k2.y() || k1.z() != k2.z() || k1.level() != k2.level();
}
inline std::ostream& operator<<(std::ostream& out, const VoxelKey& k)
{
out << k.toString();
return out;
}
inline bool operator<(const VoxelKey& k1, const VoxelKey& k2)
{
if (k1.x() != k2.x())
return k1.x() < k2.x();
if (k1.y() != k2.y())
return k1.y() < k2.y();
if (k1.z() != k2.z())
return k1.z() < k2.z();
return k1.level() < k2.level();
}
} // namespace untwine
namespace std
{
template<> struct hash<untwine::VoxelKey>
{
std::size_t operator()(const untwine::VoxelKey& k) const noexcept
{
static_assert(sizeof(size_t) > sizeof(int), "wrong assumption that size_t is 64 bits");
// For this to work well we just assume that the values are no more than than 16 bits.
size_t t = size_t(k.x()) << 48;
t |= size_t(k.y()) << 32;
t |= size_t(k.z()) << 16;
t |= size_t(k.level());
return t;
}
};
}

124
external/untwine/untwine/src/api/untwine.h vendored Normal file
View File

@ -0,0 +1,124 @@
/*****************************************************************************
* Copyright (c) 2020, Hobu, Inc. (info@hobu.co) *
* *
* All rights reserved. *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 3 of the License, or *
* (at your option) any later version. *
* *
****************************************************************************/
#ifndef UNTWINE_H
#define UNTWINE_H
#ifdef UNTWINE_STATIC
# define UNTWINE_EXPORT
#else
# if defined _WIN32 || defined __CYGWIN__
# ifdef UNTWINE_EXPORTS
# ifdef __GNUC__
# define UNTWINE_EXPORT __attribute__ ((dllexport))
# else
# define UNTWINE_EXPORT __declspec(dllexport) // Note: actually gcc seems to also supports this syntax.
# endif
# else
# ifdef __GNUC__
# define UNTWINE_EXPORT __attribute__ ((dllimport))
# else
# define UNTWINE_EXPORT __declspec(dllimport) // Note: actually gcc seems to also supports this syntax.
# endif
# endif
# else
# if __GNUC__ >= 4
# define UNTWINE_EXPORT __attribute__ ((visibility ("default")))
# else
# define UNTWINE_EXPORT
# endif
# endif
#endif
#include <functional>
#include <string>
/**
* Log levels
*/
namespace Untwine {
/**
* Returns untwine version (x.y.z)
*/
UNTWINE_EXPORT std::string version();
enum LogLevel
{
Error,
Warn,
Info,
Debug
};
typedef std::function<void(LogLevel logLevel, const std::string& message)> LoggerCallbackFunction;
/**
* Sets custom callback for logging output
*/
UNTWINE_EXPORT void SetLoggerCallback( LoggerCallbackFunction callback );
/**
* Sets maximum log level (verbosity)
*
* By default logger outputs errors only.
* Log levels (low to high): Error, Warn, Info, Debug
* For example, if LogLevel is set to Warn, logger outputs errors and warnings.
*/
UNTWINE_EXPORT void SetLogVerbosity( LogLevel verbosity );
UNTWINE_EXPORT struct Feedback
{
enum Status {
Ready = 0, // before start
Canceled, // user cancelled, not running
Running, // running (cancellation may be already requested)
Finished, //Success (finished)
Failed // Failed to succeed (all reasons BUT cancellation by user)
};
bool cancellationRequested = false; //!< QGIS sets this, UNTWINE reads it and try to cancel the job when first possible
Status status = Ready; //! UNTWINE sets this when the status of job is changed, QGIS reads it
float progress = 0.0f; //!< 0-100, UNTWINE sets this periodically as the job progress goes. QGIS reads it.
}
/**
* Starts a new preflight step from the folder of files or a single point cloud file
* \param uri single point cloud file readable by PDAL
* \param outputDir folder to write point cloud buckets
* \param options string map defining options/flags, empty for all defaults
* \param feedback feedback for reporting progress, result of task and pass user request for cancellation
*/
UNTWINE_EXPORT void PreFlightClustering(
const std::string& uri,
const std::string& outputDir,
const std::map<std::string, std::string>& options,
Feedback& feedback
);
/**
* Starts a new bottom-up indexing
* \param inputDir input directory from UNTWINE_PreFlight
* \param outputDir folder to write EPT files
* \param options string map defining options/flags, empty for all defaults
* \param feedback feedback for reporting progress, result of task and pass user request for cancellation
*/
UNTWINE_EXPORT void BottomUpIndexing(
const std::string& inputDir,
const std::string& outputDir,
const std::map<std::string, std::string>& options,
Feedback& feedback
);
} // namespace Untwine
#endif // UNTWINE_H

26
external/untwine_to_qgis.bash vendored Executable file
View File

@ -0,0 +1,26 @@
#!/usr/bin/env bash
if [ "$#" -ne 1 ] ; then
echo "untwine_to_qgis: untwine directory argument required"
exit 1
fi
UNTWINE_QGIS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
UNTWINE_QGIS_DIR=$UNTWINE_QGIS_DIR/untwine
UNTWINE_DIR=$1
if [ ! -d "$UNTWINE_DIR/untwine" ] ; then
echo "untwine_to_qgis: Directory $UNTWINE_DIR/untwine does not exist"
exit 1
fi
PWD=`pwd`
echo "untwine_to_qgis: Remove old version"
rm -rf $UNTWINE_QGIS_DIR/*
echo "untwine_to_qgis: Copy new version"
rsync -r $UNTWINE_DIR $UNTWINE_QGIS_DIR --exclude="CMakeLists.txt*" --exclude="cmake/" --exclude="README.md" --exclude=".git" --exclude=".gitignore"
echo "untwine_to_qgis: Done"
cd $PWD

42
python/core/auto_generated/pointcloud/qgspointclouddataprovider.sip.in
View File

@ -40,6 +40,13 @@ Responsible for reading native point cloud data and returning the indexed data.
typedef QFlags<QgsPointCloudDataProvider::Capability> Capabilities;
enum PointCloudIndexGenerationState
{
NotIndexed,
Indexing,
Indexed
};
QgsPointCloudDataProvider( const QString &uri,
const QgsDataProvider::ProviderOptions &providerOptions,
QgsDataProvider::ReadFlags flags = QgsDataProvider::ReadFlags() );
@ -57,8 +64,36 @@ Returns flags containing the supported capabilities for the data provider.
virtual QgsPointCloudAttributeCollection attributes() const = 0;
%Docstring
Returns the attributes available from this data provider.
May return empty collection until :py:func:`~QgsPointCloudDataProvider.pointCloudIndexLoaded` is emitted
%End
virtual void loadIndex( ) = 0;
%Docstring
Triggers loading of the point cloud index
\sa :py:func:`~QgsPointCloudDataProvider.index`
%End
virtual void generateIndex( ) = 0;
%Docstring
Triggers generation of the point cloud index
emits :py:func:`~QgsPointCloudDataProvider.indexGenerationStateChanged`
\sa :py:func:`~QgsPointCloudDataProvider.index`
%End
virtual PointCloudIndexGenerationState indexingState( ) = 0;
%Docstring
Gets the current index generation state
%End
bool hasValidIndex() const;
%Docstring
Returns whether provider has index which is valid
%End
virtual int pointCount() const = 0;
%Docstring
@ -183,7 +218,6 @@ Lidar Point Classes.
.. seealso:: :py:func:`lasClassificationCodes`
%End
static QMap< int, QString > dataFormatIds();
%Docstring
Returns the map of LAS data format ID to untranslated string value.
@ -198,6 +232,12 @@ Returns the map of LAS data format ID to translated string value.
.. seealso:: :py:func:`dataFormatIds`
%End
signals:
void indexGenerationStateChanged( PointCloudIndexGenerationState state );
%Docstring
Emitted when point cloud generation state is changed
%End
};
/************************************************************************

2
python/core/auto_generated/pointcloud/qgspointcloudlayer.sip.in
View File

@ -43,6 +43,8 @@ Constructor for LayerOptions with optional ``transformContext``.
bool loadDefaultStyle;
bool skipCrsValidation;
bool skipIndexGeneration;
};

2
scripts/astyle.sh
View File

@ -105,7 +105,7 @@ astyleit() {
for f in "$@"; do
case "$f" in
src/plugins/grass/qtermwidget/*|external/qwt*|external/o2/*|external/qt-unix-signals/*|external/rtree/*|external/astyle/*|external/kdbush/*|external/poly2tri/*|external/wintoast/*|external/qt3dextra-headers/*|external/laz-perf/*|external/meshOptimizer/*|external/mapbox-vector-tile/*|python/ext-libs/*|ui_*.py|*.astyle|tests/testdata/*|editors/*)
src/plugins/grass/qtermwidget/*|external/untwine/*|external/qwt*|external/o2/*|external/qt-unix-signals/*|external/rtree/*|external/astyle/*|external/kdbush/*|external/poly2tri/*|external/wintoast/*|external/qt3dextra-headers/*|external/laz-perf/*|external/meshOptimizer/*|external/mapbox-vector-tile/*|python/ext-libs/*|ui_*.py|*.astyle|tests/testdata/*|editors/*)
echo -ne "$f skipped $elcr"
continue
;;

8
src/core/CMakeLists.txt
View File

@ -1704,14 +1704,14 @@ if (WITH_EPT)
set(QGIS_CORE_SRCS ${QGIS_CORE_SRCS}
providers/ept/qgseptdataitems.cpp
providers/ept/qgseptprovider.cpp
providers/ept/qgseptdecoder.cpp
providers/ept/qgseptpointcloudindex.cpp
pointcloud/qgseptdecoder.cpp
pointcloud/qgseptpointcloudindex.cpp
)
set(QGIS_CORE_HDRS ${QGIS_CORE_HDRS}
providers/ept/qgseptdataitems.h
providers/ept/qgseptprovider.h
providers/ept/qgseptdecoder.h
providers/ept/qgseptpointcloudindex.h
pointcloud/qgseptdecoder.h
pointcloud/qgseptpointcloudindex.h
)
endif()

3
src/core/providers/ept/qgseptdecoder.cpp → src/core/pointcloud/qgseptdecoder.cpp
View File

@ -262,7 +262,8 @@ QgsPointCloudBlock *QgsEptDecoder::decompressLaz( const QString &filename,
{
Q_UNUSED( attributes )
std::ifstream file( filename.toLatin1().constData(), std::ios::binary );
const QByteArray arr = filename.toUtf8();
std::ifstream file( arr.constData(), std::ios::binary );
if ( ! file.good() )
return nullptr;

0
src/core/providers/ept/qgseptdecoder.h → src/core/pointcloud/qgseptdecoder.h
View File

29
src/core/providers/ept/qgseptpointcloudindex.cpp → src/core/pointcloud/qgseptpointcloudindex.cpp
View File

@ -31,6 +31,8 @@
#include "qgspointcloudrequest.h"
#include "qgspointcloudattribute.h"
#include "qgslogger.h"
#include "qgsfeedback.h"
#include "qgsmessagelog.h"
///@cond PRIVATE
@ -41,16 +43,29 @@ QgsEptPointCloudIndex::QgsEptPointCloudIndex() = default;
QgsEptPointCloudIndex::~QgsEptPointCloudIndex() = default;
bool QgsEptPointCloudIndex::load( const QString &fileName )
void QgsEptPointCloudIndex::load( const QString &fileName )
{
// mDirectory = directory;
QFile f( fileName );
if ( !f.open( QIODevice::ReadOnly ) )
return false;
{
QgsMessageLog::logMessage( tr( "Unable to open %1 for reading" ).arg( fileName ) );
mIsValid = false;
return;
}
const QDir directory = QFileInfo( fileName ).absoluteDir();
mDirectory = directory.absolutePath();
bool success = loadSchema( f );
if ( success )
{
success = loadHierarchy();
}
mIsValid = success;
}
bool QgsEptPointCloudIndex::loadSchema( QFile &f )
{
QByteArray dataJson = f.readAll();
QJsonParseError err;
QJsonDocument doc = QJsonDocument::fromJson( dataJson, &err );
@ -243,8 +258,7 @@ bool QgsEptPointCloudIndex::load( const QString &fileName )
QgsDebugMsgLevel( QStringLiteral( "res at lvl2 %1 with node size %2" ).arg( dx / mSpan / 4 ).arg( dx / 4 ), 2 );
#endif
// load hierarchy
return loadHierarchy();
return true;
}
QgsPointCloudBlock *QgsEptPointCloudIndex::nodeData( const IndexedPointCloudNode &n, const QgsPointCloudRequest &request )
@ -391,4 +405,9 @@ bool QgsEptPointCloudIndex::loadHierarchy()
return true;
}
bool QgsEptPointCloudIndex::isValid() const
{
return mIsValid;
}
///@endcond

8
src/core/providers/ept/qgseptpointcloudindex.h → src/core/pointcloud/qgseptpointcloudindex.h
View File

@ -24,6 +24,7 @@
#include <QStringList>
#include <QVector>
#include <QList>
#include <QFile>
#include "qgspointcloudindex.h"
#include "qgspointcloudattribute.h"
@ -35,7 +36,7 @@
class QgsCoordinateReferenceSystem;
class QgsEptPointCloudIndex: public QgsPointCloudIndex
class CORE_EXPORT QgsEptPointCloudIndex: public QgsPointCloudIndex
{
Q_OBJECT
public:
@ -43,7 +44,7 @@ class QgsEptPointCloudIndex: public QgsPointCloudIndex
explicit QgsEptPointCloudIndex();
~QgsEptPointCloudIndex();
bool load( const QString &fileName ) override;
void load( const QString &fileName ) override;
QgsPointCloudBlock *nodeData( const IndexedPointCloudNode &n, const QgsPointCloudRequest &request ) override;
@ -54,10 +55,13 @@ class QgsEptPointCloudIndex: public QgsPointCloudIndex
QVariant metadataClassStatistic( const QString &attribute, const QVariant &value, QgsStatisticalSummary::Statistic statistic ) const;
QVariantMap originalMetadata() const { return mOriginalMetadata; }
bool isValid() const override;
private:
bool loadSchema( QFile &f );
bool loadHierarchy();
bool mIsValid = false;
QString mDataType;
QString mDirectory;
QString mWkt;

5
src/core/pointcloud/qgspointclouddataprovider.cpp
View File

@ -36,6 +36,11 @@ QgsPointCloudDataProvider::Capabilities QgsPointCloudDataProvider::capabilities(
return QgsPointCloudDataProvider::NoCapabilities;
}
bool QgsPointCloudDataProvider::hasValidIndex() const
{
return index() && index()->isValid();
}
QgsGeometry QgsPointCloudDataProvider::polygonBounds() const
{
return QgsGeometry::fromRect( extent() );

45
src/core/pointcloud/qgspointclouddataprovider.h
View File

@ -56,6 +56,16 @@ class CORE_EXPORT QgsPointCloudDataProvider: public QgsDataProvider
Q_DECLARE_FLAGS( Capabilities, Capability )
/**
* Point cloud index state
*/
enum PointCloudIndexGenerationState
{
NotIndexed = 0, //!< Provider has no index available
Indexing = 1 << 0, //!< Provider try to index the source data
Indexed = 1 << 1 //!< The index is ready to be used
};
//! Ctor
QgsPointCloudDataProvider( const QString &uri,
const QgsDataProvider::ProviderOptions &providerOptions,
@ -70,9 +80,32 @@ class CORE_EXPORT QgsPointCloudDataProvider: public QgsDataProvider
/**
* Returns the attributes available from this data provider.
* May return empty collection until pointCloudIndexLoaded() is emitted
*/
virtual QgsPointCloudAttributeCollection attributes() const = 0;
/**
* Triggers loading of the point cloud index
*
* \sa index()
*/
virtual void loadIndex( ) = 0;
/**
* Triggers generation of the point cloud index
*
* emits indexGenerationStateChanged()
*
* \sa index()
*/
virtual void generateIndex( ) = 0;
/**
* Gets the current index generation state
*/
virtual PointCloudIndexGenerationState indexingState( ) = 0;
/**
* Returns the point cloud index associated with the provider.
*
@ -82,6 +115,11 @@ class CORE_EXPORT QgsPointCloudDataProvider: public QgsDataProvider
*/
virtual QgsPointCloudIndex *index() const SIP_SKIP {return nullptr;}
/**
* Returns whether provider has index which is valid
*/
bool hasValidIndex() const;
/**
* Returns the total number of points available in the dataset.
*/
@ -235,7 +273,6 @@ class CORE_EXPORT QgsPointCloudDataProvider: public QgsDataProvider
*/
static QMap< int, QString > translatedLasClassificationCodes();
/**
* Returns the map of LAS data format ID to untranslated string value.
*
@ -250,6 +287,12 @@ class CORE_EXPORT QgsPointCloudDataProvider: public QgsDataProvider
*/
static QMap< int, QString > translatedDataFormatIds();
signals:
/**
* Emitted when point cloud generation state is changed
*/
void indexGenerationStateChanged( PointCloudIndexGenerationState state );
};
#endif // QGSMESHDATAPROVIDER_H

5
src/core/pointcloud/qgspointcloudindex.h
View File

@ -149,7 +149,10 @@ class CORE_EXPORT QgsPointCloudIndex: public QObject
~QgsPointCloudIndex();
//! Loads the index from the file
virtual bool load( const QString &fileName ) = 0;
virtual void load( const QString &fileName ) = 0;
//! Returns whether index is loaded and valid
virtual bool isValid() const = 0;
//! Returns root node of the index
IndexedPointCloudNode root() { return IndexedPointCloudNode( 0, 0, 0, 0 ); }

30
src/core/pointcloud/qgspointcloudlayer.cpp
View File

@ -38,10 +38,14 @@ QgsPointCloudLayer::QgsPointCloudLayer( const QString &path,
: QgsMapLayer( QgsMapLayerType::PointCloudLayer, baseName, path )
, mElevationProperties( new QgsPointCloudLayerElevationProperties( this ) )
{
if ( !path.isEmpty() && !providerLib.isEmpty() )
{
QgsDataProvider::ProviderOptions providerOptions { options.transformContext };
setDataSource( path, baseName, providerLib, providerOptions, options.loadDefaultStyle );
if ( !options.skipIndexGeneration && mDataProvider && mDataProvider->isValid() )
mDataProvider.get()->generateIndex();
}
setLegend( QgsMapLayerLegend::defaultPointCloudLegend( this ) );
@ -273,10 +277,14 @@ void QgsPointCloudLayer::setTransformContext( const QgsCoordinateTransformContex
mDataProvider->setTransformContext( transformContext );
}
void QgsPointCloudLayer::setDataSource( const QString &dataSource, const QString &baseName, const QString &provider, const QgsDataProvider::ProviderOptions &options, bool loadDefaultStyleFlag )
void QgsPointCloudLayer::setDataSource( const QString &dataSource, const QString &baseName, const QString &provider,
const QgsDataProvider::ProviderOptions &options, bool loadDefaultStyleFlag )
{
if ( mDataProvider )
{
disconnect( mDataProvider.get(), &QgsPointCloudDataProvider::dataChanged, this, &QgsPointCloudLayer::dataChanged );
disconnect( mDataProvider.get(), &QgsPointCloudDataProvider::indexGenerationStateChanged, this, &QgsPointCloudLayer::onPointCloudIndexGenerationStateChanged );
}
setName( baseName );
mProviderKey = provider;
@ -308,7 +316,12 @@ void QgsPointCloudLayer::setDataSource( const QString &dataSource, const QString
return;
}
connect( mDataProvider.get(), &QgsPointCloudDataProvider::indexGenerationStateChanged, this, &QgsPointCloudLayer::onPointCloudIndexGenerationStateChanged );
connect( mDataProvider.get(), &QgsPointCloudDataProvider::dataChanged, this, &QgsPointCloudLayer::dataChanged );
// Load initial extent, crs and renderer
setCrs( mDataProvider->crs() );
setExtent( mDataProvider->extent() );
if ( !mRenderer || loadDefaultStyleFlag )
{
@ -341,12 +354,23 @@ void QgsPointCloudLayer::setDataSource( const QString &dataSource, const QString
}
}
connect( mDataProvider.get(), &QgsPointCloudDataProvider::dataChanged, this, &QgsPointCloudLayer::dataChanged );
emit dataSourceChanged();
triggerRepaint();
}
void QgsPointCloudLayer::onPointCloudIndexGenerationStateChanged( QgsPointCloudDataProvider::PointCloudIndexGenerationState state )
{
if ( state == QgsPointCloudDataProvider::Indexed )
{
mDataProvider.get()->loadIndex();
if ( mRenderer->type() == QLatin1String( "extent" ) )
{
setRenderer( QgsApplication::pointCloudRendererRegistry()->defaultRenderer( mDataProvider.get() ) );
}
triggerRepaint();
}
}
QString QgsPointCloudLayer::loadDefaultStyle( bool &resultFlag )
{
if ( mDataProvider->capabilities() & QgsPointCloudDataProvider::CreateRenderer )

9
src/core/pointcloud/qgspointcloudlayer.h
View File

@ -77,6 +77,11 @@ class CORE_EXPORT QgsPointCloudLayer : public QgsMapLayer
* layer.
*/
bool skipCrsValidation = false;
/**
* Set to TRUE if point cloud index generation should be skipped.
*/
bool skipIndexGeneration = false;
};
@ -162,6 +167,9 @@ class CORE_EXPORT QgsPointCloudLayer : public QgsMapLayer
*/
void setRenderer( QgsPointCloudRenderer *renderer SIP_TRANSFER );
private slots:
void onPointCloudIndexGenerationStateChanged( QgsPointCloudDataProvider::PointCloudIndexGenerationState state );
private:
bool isReadOnly() const override {return true;}
@ -175,7 +183,6 @@ class CORE_EXPORT QgsPointCloudLayer : public QgsMapLayer
std::unique_ptr<QgsPointCloudRenderer> mRenderer;
QgsPointCloudLayerElevationProperties *mElevationProperties = nullptr;
};

2
src/core/pointcloud/qgspointcloudlayerrenderer.cpp
View File

@ -70,7 +70,7 @@ bool QgsPointCloudLayerRenderer::render()
// TODO cache!?
QgsPointCloudIndex *pc = mLayer->dataProvider()->index();
if ( !pc )
if ( !pc || !pc->isValid() )
return false;
mRenderer->startRender( context );

2
src/core/pointcloud/qgspointcloudrendererregistry.cpp
View File

@ -87,7 +87,7 @@ QgsPointCloudRenderer *QgsPointCloudRendererRegistry::defaultRenderer( const Qgs
if ( !provider )
return new QgsPointCloudAttributeByRampRenderer();
if ( provider->name() == QLatin1String( "pdal" ) )
if ( ( provider->name() == QLatin1String( "pdal" ) ) && ( !provider->hasValidIndex() ) )
{
// for now, default to extent renderer only for las/laz files
return new QgsPointCloudExtentRenderer();

17
src/core/providers/ept/qgseptprovider.cpp
View File

@ -38,7 +38,7 @@ QgsEptProvider::QgsEptProvider(
if ( QgsApplication::profiler()->groupIsActive( QStringLiteral( "projectload" ) ) )
profile = qgis::make_unique< QgsScopedRuntimeProfile >( tr( "Open data source" ), QStringLiteral( "projectload" ) );
mIsValid = mIndex->load( uri );
loadIndex( );
}
QgsEptProvider::~QgsEptProvider() = default;
@ -60,7 +60,7 @@ QgsPointCloudAttributeCollection QgsEptProvider::attributes() const
bool QgsEptProvider::isValid() const
{
return mIsValid;
return mIndex->isValid();
}
QString QgsEptProvider::name() const
@ -93,11 +93,24 @@ QVariant QgsEptProvider::metadataClassStatistic( const QString &attribute, const
return mIndex->metadataClassStatistic( attribute, value, statistic );
}
void QgsEptProvider::loadIndex( )
{
if ( mIndex->isValid() )
return;
mIndex->load( dataSourceUri() );
}
QVariantMap QgsEptProvider::originalMetadata() const
{
return mIndex->originalMetadata();
}
void QgsEptProvider::generateIndex()
{
//no-op, index is always generated
}
QVariant QgsEptProvider::metadataStatistic( const QString &attribute, QgsStatisticalSummary::Statistic statistic ) const
{
return mIndex->metadataStatistic( attribute, statistic );

8
src/core/providers/ept/qgseptprovider.h
View File

@ -44,23 +44,21 @@ class QgsEptProvider: public QgsPointCloudDataProvider
QgsRectangle extent() const override;
QgsPointCloudAttributeCollection attributes() const override;
bool isValid() const override;
QString name() const override;
QString description() const override;
QgsPointCloudIndex *index() const override;
int pointCount() const override;
QVariant metadataStatistic( const QString &attribute, QgsStatisticalSummary::Statistic statistic ) const override;
QVariantList metadataClasses( const QString &attribute ) const override;
QVariant metadataClassStatistic( const QString &attribute, const QVariant &value, QgsStatisticalSummary::Statistic statistic ) const override;
QVariantMap originalMetadata() const override;
void loadIndex( ) override;
void generateIndex( ) override;
PointCloudIndexGenerationState indexingState( ) override { return PointCloudIndexGenerationState::Indexed; }
private:
std::unique_ptr<QgsEptPointCloudIndex> mIndex;
bool mIsValid = false;
};
class QgsEptProviderMetadata : public QgsProviderMetadata

115
src/providers/pdal/CMakeLists.txt
View File

@ -1,30 +1,88 @@
########################################################
# Files
find_package(Threads REQUIRED)
set(PDAL_SRCS
qgspdaldataitems.cpp
qgspdalprovider.cpp
qgspdaleptgenerationtask.cpp
${CMAKE_SOURCE_DIR}/external/untwine/api/QgisUntwine.cpp
)
set(PDAL_HDRS
qgspdaldataitems.h
qgspdalprovider.h
qgspdaleptgenerationtask.h
${CMAKE_SOURCE_DIR}/external/untwine/api/QgisUntwine.hpp
)
if (WITH_GUI)
set(PDAL_SRCS
${PDAL_SRCS}
set(PDAL_GUI_SRCS
qgspdalprovidergui.cpp
qgspdaldataitemguiprovider.cpp
)
set(PDAL_HDRS
${PDAL_HDRS}
set(PDAL_GUI_HDRS
qgspdalprovidergui.h
qgspdaldataitemguiprovider.h
)
endif()
set(UNTWINE_SRCS
${CMAKE_SOURCE_DIR}/external/untwine/bu/BuPyramid.cpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/Processor.cpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/PyramidManager.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/BufferCache.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Cell.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Epf.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/FileProcessor.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Grid.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Reprocessor.cpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Writer.cpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/ProgressWriter.cpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/Untwine.cpp
)
set(UNTWINE_HDRS
${CMAKE_SOURCE_DIR}/external/untwine/bu/BuPyramid.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/BuTypes.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/FileInfo.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/OctantInfo.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/PointAccessor.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/Processor.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/PyramidManager.hpp
${CMAKE_SOURCE_DIR}/external/untwine/bu/VoxelInfo.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/BufferCache.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Cell.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Epf.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/EpfTypes.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/FileProcessor.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Grid.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Reprocessor.hpp
${CMAKE_SOURCE_DIR}/external/untwine/epf/Writer.hpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/Common.hpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/FileDimInfo.hpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/GridKey.hpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/Point.hpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/ProgressWriter.hpp
${CMAKE_SOURCE_DIR}/external/untwine/untwine/VoxelKey.hpp
)
configure_file(${CMAKE_SOURCE_DIR}/external/untwine/untwine/Config.hpp.in ${CMAKE_BINARY_DIR}/untwine/Config.hpp)
set(UNTWINE_INCLUDE_DIRS
${CMAKE_SOURCE_DIR}/external/untwine/untwine
${CMAKE_SOURCE_DIR}/external/untwine/epf
${CMAKE_SOURCE_DIR}/external/untwine/bu
${CMAKE_SOURCE_DIR}/untwine/api
${CMAKE_BINARY_DIR}/untwine
)
########################################################
# Build
@ -43,6 +101,7 @@ include_directories(
${CMAKE_SOURCE_DIR}/src/gui/codeeditors
${CMAKE_SOURCE_DIR}/external
${CMAKE_SOURCE_DIR}/external/nlohmann
${CMAKE_SOURCE_DIR}/external/untwine/api
${CMAKE_BINARY_DIR}/src/core
${CMAKE_BINARY_DIR}/src/gui
@ -53,10 +112,35 @@ include_directories(SYSTEM
${PDAL_INCLUDE_DIR}
)
add_library (pdalprovider MODULE ${PDAL_SRCS} ${PDAL_HDRS})
target_link_libraries (pdalprovider
add_executable(untwine ${UNTWINE_SRCS} ${UNTWINE_HDRS})
set_target_properties(untwine PROPERTIES
RUNTIME_OUTPUT_DIRECTORY ${QGIS_OUTPUT_DIRECTORY}/${QGIS_LIBEXEC_SUBDIR}
)
target_link_libraries (untwine
${PDAL_LIBRARIES}
)
target_include_directories(untwine PRIVATE ${UNTWINE_INCLUDE_DIRS})
add_library (pdalprovider MODULE ${PDAL_SRCS} ${PDAL_HDRS} ${PDAL_GUI_SRCS} ${PDAL_GUI_HDRS})
target_link_libraries (pdalprovider
${PDAL_LIBRARIES}
${CMAKE_THREAD_LIBS_INIT}
)
if (WITH_GUI)
target_link_libraries (pdalprovider
${PDAL_LIBRARIES}
${Qt5Xml_LIBRARIES}
${Qt5Core_LIBRARIES}
${Qt5Svg_LIBRARIES}
${Qt5Network_LIBRARIES}
${Qt5Sql_LIBRARIES}
${Qt5Concurrent_LIBRARIES}
)
target_link_libraries(pdalprovider qgis_gui)
add_dependencies(pdalprovider ui)
endif()
# static library
add_library (pdalprovider_a STATIC ${PDAL_SRCS} ${PDAL_HDRS})
@ -71,12 +155,25 @@ target_link_libraries (pdalprovider_a
)
if (WITH_GUI)
target_link_libraries(pdalprovider
qgis_gui
add_library (pdalprovider_gui_a STATIC ${PDAL_GUI_SRCS} ${PDAL_GUI_HDRS})
target_link_libraries (pdalprovider_gui_a
${PDAL_LIBRARIES}
${Qt5Xml_LIBRARIES}
${Qt5Core_LIBRARIES}
${Qt5Svg_LIBRARIES}
${Qt5Network_LIBRARIES}
${Qt5Sql_LIBRARIES}
${Qt5Concurrent_LIBRARIES}
)
add_dependencies(pdalprovider ui)
target_link_libraries(pdalprovider_gui_a qgis_gui)
add_dependencies(pdalprovider_gui_a ui)
endif()
install(TARGETS pdalprovider
RUNTIME DESTINATION ${QGIS_PLUGIN_DIR}
LIBRARY DESTINATION ${QGIS_PLUGIN_DIR})
# override default path where built files are put to allow running qgis without installing
install(TARGETS untwine
RUNTIME DESTINATION ${QGIS_LIBEXEC_DIR}
PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)

181
src/providers/pdal/qgspdaleptgenerationtask.cpp Normal file
View File

@ -0,0 +1,181 @@
/***************************************************************************
qgspdaleptgenerationtask.cpp
------------------------
Date : December 2020
Copyright : (C) 2020 by Peter Petrik
Email : zilolv at gmail dot com
***************************************************************************
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 2 of the License, or *
* (at your option) any later version. *
* *
***************************************************************************/
#include "qgspdaleptgenerationtask.h"
#include <vector>
#include <string>
#include <QDebug>
#include <QThread>
#include <QFileInfo>
#include <QDir>
#include <QProcessEnvironment>
#include "qgsapplication.h"
#include "QgisUntwine.hpp"
#include "qgsmessagelog.h"
#include "qgis.h"
QgsPdalEptGenerationTask::QgsPdalEptGenerationTask( const QString &file, const QString &outputDir )
: QgsTask( tr( "Generate EPT Index" ) )
, mOutputDir( outputDir )
, mFile( file )
{
mUntwineExecutableBinary = guessUntwineExecutableBinary();
}
bool QgsPdalEptGenerationTask::run()
{
if ( isCanceled() || !prepareOutputDir() )
return false;
if ( isCanceled() || !runUntwine() )
return false;
if ( isCanceled() )
return false;
cleanTemp();
return true;
}
void QgsPdalEptGenerationTask::cleanTemp()
{
QDir tmpDir( mOutputDir + QStringLiteral( "/temp" ) );
if ( tmpDir.exists() )
{
QgsMessageLog::logMessage( tr( "Removing temporary files in %1" ).arg( tmpDir.dirName() ), QObject::tr( "Point clouds" ), Qgis::Info );
tmpDir.removeRecursively();
}
}
bool QgsPdalEptGenerationTask::runUntwine()
{
QFileInfo executable( mUntwineExecutableBinary );
if ( !executable.isExecutable() )
{
QgsMessageLog::logMessage( tr( "Untwine executable not found %1" ).arg( mUntwineExecutableBinary ), QObject::tr( "Point clouds" ), Qgis::Critical );
return false;
}
else
{
QgsMessageLog::logMessage( tr( "Using executable %1" ).arg( mUntwineExecutableBinary ), QObject::tr( "Point clouds" ), Qgis::Info );
}
untwine::QgisUntwine untwineProcess( mUntwineExecutableBinary.toStdString() );
std::vector<std::string> files = {mFile.toStdString()};
untwineProcess.start( files, mOutputDir.toStdString() );
int lastPercent = 0;
while ( true )
{
QThread::msleep( 100 );
int percent = untwineProcess.progressPercent();
if ( lastPercent != percent )
{
QString message = QString::fromStdString( untwineProcess.progressMessage() );
if ( !message.isEmpty() )
QgsMessageLog::logMessage( message, QObject::tr( "Point clouds" ), Qgis::Info );
setProgress( percent );
}
if ( isCanceled() )
{
untwineProcess.stop();
return false;
}
if ( !untwineProcess.running() )
{
setProgress( 100 );
return true;
}
}
}
QString QgsPdalEptGenerationTask::untwineExecutableBinary() const
{
return mUntwineExecutableBinary;
}
void QgsPdalEptGenerationTask::setUntwineExecutableBinary( const QString &untwineExecutableBinary )
{
mUntwineExecutableBinary = untwineExecutableBinary;
}
QString QgsPdalEptGenerationTask::guessUntwineExecutableBinary() const
{
QString untwineExecutable = QProcessEnvironment::systemEnvironment().value( QStringLiteral( "QGIS_UNTWINE_EXECUTABLE" ) );
if ( untwineExecutable.isEmpty() )
{
#if defined(Q_OS_WIN)
untwineExecutable = QgsApplication::libexecPath() + "untwine.exe";
#else
untwineExecutable = QgsApplication::libexecPath() + "untwine";
#endif
}
return QString( untwineExecutable );
}
QString QgsPdalEptGenerationTask::outputDir() const
{
return mOutputDir;
}
bool QgsPdalEptGenerationTask::prepareOutputDir()
{
QFileInfo fi( mOutputDir + "/ept.json" );
if ( fi.exists() )
{
QgsMessageLog::logMessage( tr( "File %1 is already indexed" ).arg( mFile ), QObject::tr( "Point clouds" ), Qgis::Info );
return true;
}
else
{
if ( QDir( mOutputDir ).exists() )
{
if ( !QDir( mOutputDir ).isEmpty() )
{
if ( QDir( mOutputDir + "/temp" ).exists() )
{
QgsMessageLog::logMessage( tr( "Another indexing process is running (or finished with crash) in directory %1" ).arg( mOutputDir ), QObject::tr( "Point clouds" ), Qgis::Warning );
return false;
}
else
{
QgsMessageLog::logMessage( tr( "Folder %1 is non-empty, but there isn't ept.json present." ).arg( mOutputDir ), QObject::tr( "Point clouds" ), Qgis::Critical );
return false;
}
}
}
else
{
bool success = QDir().mkdir( mOutputDir );
if ( success )
{
QgsMessageLog::logMessage( tr( "Created output directory %1" ).arg( mOutputDir ), QObject::tr( "Point clouds" ), Qgis::Info );
}
else
{
QgsMessageLog::logMessage( tr( "Unable to create output directory %1" ).arg( mOutputDir ), QObject::tr( "Point clouds" ), Qgis::Critical );
return false;
}
}
}
return true;
}

46
src/providers/pdal/qgspdaleptgenerationtask.h Normal file
View File

@ -0,0 +1,46 @@
/***************************************************************************
qgspdaleptgenerationtask.h
------------------------
Date : December 2020
Copyright : (C) 2020 by Peter Petrik
Email : zilolv at gmail dot com
***************************************************************************
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 2 of the License, or *
* (at your option) any later version. *
* *
***************************************************************************/
#ifndef QGSPDALEPTGENERATIONTASK_H
#define QGSPDALEPTGENERATIONTASK_H
#include <QObject>
#include "qgstaskmanager.h"
class QgsPdalEptGenerationTask: public QgsTask
{
Q_OBJECT
public:
QgsPdalEptGenerationTask( const QString &file, const QString &outputDir );
bool run() override;
QString untwineExecutableBinary() const;
void setUntwineExecutableBinary( const QString &untwineExecutableBinary );
QString outputDir() const;
private:
bool prepareOutputDir();
bool runUntwine();
void cleanTemp();
QString guessUntwineExecutableBinary() const;
QString mUntwineExecutableBinary;
QString mOutputDir;
QString mFile;
};
#endif // QGSPDALEPTGENERATIONTASK_H

35
src/providers/pdal/qgspdalgresprovidergui.h
View File

@ -1,35 +0,0 @@
/***************************************************************************
qgspostgresprovidergui.h
------------------------
Date : October 2019
Copyright : (C) 2019 by Peter Petrik
Email : zilolv at gmail dot com
***************************************************************************
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 2 of the License, or *
* (at your option) any later version. *
* *
***************************************************************************/
#ifndef QGSPOSTGRESPROVIDERGUI_H
#define QGSPOSTGRESPROVIDERGUI_H
#include <QList>
#include <QMainWindow>
#include "qgsproviderguimetadata.h"
class QgsPostgresProviderGuiMetadata: public QgsProviderGuiMetadata
{
public:
QgsPostgresProviderGuiMetadata();
QList<QgsSourceSelectProvider *> sourceSelectProviders() override;
QList<QgsDataItemGuiProvider *> dataItemGuiProviders() override;
QList<QgsProjectStorageGuiProvider *> projectStorageGuiProviders() override;
void registerGui( QMainWindow *mainWindow ) override;
};
#endif // QGSPOSTGRESPROVIDERGUI_H

126
src/providers/pdal/qgspdalprovider.cpp
View File

@ -23,11 +23,15 @@
#include "qgslogger.h"
#include "qgsmessagelog.h"
#include "qgsjsonutils.h"
#include "qgspdaleptgenerationtask.h"
#include "qgseptpointcloudindex.h"
#include "qgstaskmanager.h"
#include <pdal/io/LasReader.hpp>
#include <pdal/io/LasHeader.hpp>
#include <pdal/Options.hpp>
#define PROVIDER_KEY QStringLiteral( "pdal" )
#define PROVIDER_DESCRIPTION QStringLiteral( "PDAL point cloud data provider" )
@ -36,12 +40,14 @@ QgsPdalProvider::QgsPdalProvider(
const QgsDataProvider::ProviderOptions &options,
QgsDataProvider::ReadFlags flags )
: QgsPointCloudDataProvider( uri, options, flags )
, mIndex( new QgsEptPointCloudIndex )
{
std::unique_ptr< QgsScopedRuntimeProfile > profile;
if ( QgsApplication::profiler()->groupIsActive( QStringLiteral( "projectload" ) ) )
profile = qgis::make_unique< QgsScopedRuntimeProfile >( tr( "Open data source" ), QStringLiteral( "projectload" ) );
mIsValid = load( uri );
loadIndex( );
}
QgsPdalProvider::~QgsPdalProvider() = default;
@ -58,8 +64,121 @@ QgsRectangle QgsPdalProvider::extent() const
QgsPointCloudAttributeCollection QgsPdalProvider::attributes() const
{
// TODO
return QgsPointCloudAttributeCollection();
return mIndex->attributes();
}
QVariantList QgsPdalProvider::metadataClasses( const QString &attribute ) const
{
return mIndex->metadataClasses( attribute );
}
QVariant QgsPdalProvider::metadataClassStatistic( const QString &attribute, const QVariant &value, QgsStatisticalSummary::Statistic statistic ) const
{
return mIndex->metadataClassStatistic( attribute, value, statistic );
}
static QString _outdir( const QString &filename )
{
const QFileInfo fi( filename );
const QDir directory = fi.absoluteDir();
const QString outputDir = QStringLiteral( "%1/ept_%2" ).arg( directory.absolutePath() ).arg( fi.baseName() );
return outputDir;
}
void QgsPdalProvider::generateIndex()
{
if ( mRunningIndexingTask || mIndex->isValid() )
return;
if ( anyIndexingTaskExists() )
{
QgsMessageLog::logMessage( tr( "EPT generation task is already running" ), QObject::tr( "Point clouds" ), Qgis::Info );
return;
}
const QString outputDir = _outdir( dataSourceUri() );
QgsPdalEptGenerationTask *generationTask = new QgsPdalEptGenerationTask( dataSourceUri(), outputDir );
connect( generationTask, &QgsPdalEptGenerationTask::taskTerminated, this, &QgsPdalProvider::onGenerateIndexFailed );
connect( generationTask, &QgsPdalEptGenerationTask::taskCompleted, this, &QgsPdalProvider::onGenerateIndexFinished );
mRunningIndexingTask = generationTask;
QgsDebugMsgLevel( "Ept Generation Task Created", 2 );
emit indexGenerationStateChanged( PointCloudIndexGenerationState::Indexing );
QgsApplication::taskManager()->addTask( generationTask );
}
QgsPointCloudDataProvider::PointCloudIndexGenerationState QgsPdalProvider::indexingState()
{
if ( mIndex->isValid() )
return PointCloudIndexGenerationState::Indexed;
else if ( mRunningIndexingTask )
return PointCloudIndexGenerationState::Indexing;
else
return PointCloudIndexGenerationState::NotIndexed;
}
void QgsPdalProvider::loadIndex( )
{
if ( mIndex->isValid() )
return;
const QString outputDir = _outdir( dataSourceUri() );
QString outEptJson = QStringLiteral( "%1/ept.json" ).arg( outputDir );
QFileInfo fi( outEptJson );
if ( fi.isFile() )
{
mIndex->load( outEptJson );
}
else
{
QgsDebugMsgLevel( QStringLiteral( "pdalprovider: ept index %1 is not correctly loaded" ).arg( outEptJson ), 2 );
}
}
void QgsPdalProvider::onGenerateIndexFinished()
{
QgsPdalEptGenerationTask *task = qobject_cast<QgsPdalEptGenerationTask *>( QObject::sender() );
// this may be already canceled task that we don't care anymore...
if ( task == mRunningIndexingTask )
{
mRunningIndexingTask = nullptr;
emit indexGenerationStateChanged( PointCloudIndexGenerationState::Indexed );
}
}
void QgsPdalProvider::onGenerateIndexFailed()
{
QgsPdalEptGenerationTask *task = qobject_cast<QgsPdalEptGenerationTask *>( QObject::sender() );
// this may be already canceled task that we don't care anymore...
if ( task == mRunningIndexingTask )
{
mRunningIndexingTask = nullptr;
emit indexGenerationStateChanged( PointCloudIndexGenerationState::NotIndexed );
}
}
bool QgsPdalProvider::anyIndexingTaskExists()
{
const QList< QgsTask * > tasks = QgsApplication::taskManager()->activeTasks();
for ( const QgsTask *task : tasks )
{
const QgsPdalEptGenerationTask *eptTask = qobject_cast<const QgsPdalEptGenerationTask *>( task );
if ( eptTask )
{
return true;
}
}
return false;
}
QVariant QgsPdalProvider::metadataStatistic( const QString &attribute, QgsStatisticalSummary::Statistic statistic ) const
{
if ( mIndex )
return mIndex->metadataStatistic( attribute, statistic );
else
return QVariant();
}
int QgsPdalProvider::pointCount() const
@ -89,8 +208,7 @@ QString QgsPdalProvider::description() const
QgsPointCloudIndex *QgsPdalProvider::index() const
{
// TODO automatically generate EPT index
return nullptr;
return mIndex.get();
}
bool QgsPdalProvider::load( const QString &uri )

23
src/providers/pdal/qgspdalprovider.h
View File

@ -21,9 +21,11 @@
#include "qgis_core.h"
#include "qgspointclouddataprovider.h"
#include "qgsprovidermetadata.h"
#include <memory>
class QgsEptPointCloudIndex;
class QgsPdalEptGenerationTask;
class QgsPdalProvider: public QgsPointCloudDataProvider
{
Q_OBJECT
@ -34,27 +36,36 @@ class QgsPdalProvider: public QgsPointCloudDataProvider
~QgsPdalProvider();
QgsCoordinateReferenceSystem crs() const override;
QgsRectangle extent() const override;
QgsPointCloudAttributeCollection attributes() const override;
int pointCount() const override;
QVariantMap originalMetadata() const override;
bool isValid() const override;
QString name() const override;
QString description() const override;
QgsPointCloudIndex *index() const override;
QVariant metadataStatistic( const QString &attribute, QgsStatisticalSummary::Statistic statistic ) const override;
QVariantList metadataClasses( const QString &attribute ) const override;
QVariant metadataClassStatistic( const QString &attribute, const QVariant &value, QgsStatisticalSummary::Statistic statistic ) const override;
void loadIndex( ) override;
void generateIndex( ) override;
PointCloudIndexGenerationState indexingState( ) override;
private slots:
void onGenerateIndexFinished();
void onGenerateIndexFailed();
private:
bool anyIndexingTaskExists();
bool load( const QString &uri );
QgsCoordinateReferenceSystem mCrs;
QgsRectangle mExtent;
bool mIsValid = false;
int mPointCount = 0;
QVariantMap mOriginalMetadata;
std::unique_ptr<QgsEptPointCloudIndex> mIndex;
QgsPdalEptGenerationTask *mRunningIndexingTask = nullptr;
};
class QgsPdalProviderMetadata : public QgsProviderMetadata

40
tests/src/providers/testqgspdalprovider.cpp
View File

@ -22,6 +22,7 @@
#include <QApplication>
#include <QFileInfo>
#include <QDir>
#include <QTemporaryDir>
//qgis includes...
#include "qgis.h"
@ -30,6 +31,7 @@
#include "qgspdalprovider.h"
#include "qgsmaplayer.h"
#include "qgspointcloudlayer.h"
#include "qgspdaleptgenerationtask.h"
/**
* \ingroup UnitTests
@ -52,6 +54,8 @@ class TestQgsPdalProvider : public QObject
void preferredUri();
void brokenPath();
void validLayer();
void testEptGeneration();
void testEptGenerationNonASCII();
private:
QString mTestDataDir;
@ -156,13 +160,24 @@ void TestQgsPdalProvider::preferredUri()
void TestQgsPdalProvider::brokenPath()
{
// test loading a bad layer URI
std::unique_ptr< QgsPointCloudLayer > layer = qgis::make_unique< QgsPointCloudLayer >( QStringLiteral( "not valid" ), QStringLiteral( "layer" ), QStringLiteral( "pdal" ) );
std::unique_ptr< QgsPointCloudLayer > layer = qgis::make_unique< QgsPointCloudLayer >(
QStringLiteral( "not valid" ),
QStringLiteral( "layer" ),
QStringLiteral( "pdal" ) );
QVERIFY( !layer->isValid() );
}
void TestQgsPdalProvider::validLayer()
{
std::unique_ptr< QgsPointCloudLayer > layer = qgis::make_unique< QgsPointCloudLayer >( mTestDataDir + QStringLiteral( "point_clouds/las/cloud.las" ), QStringLiteral( "layer" ), QStringLiteral( "pdal" ) );
QgsPointCloudLayer::LayerOptions options;
options.skipIndexGeneration = true;
std::unique_ptr< QgsPointCloudLayer > layer = qgis::make_unique< QgsPointCloudLayer >(
mTestDataDir + QStringLiteral( "point_clouds/las/cloud.las" ),
QStringLiteral( "layer" ),
QStringLiteral( "pdal" ),
options
);
QVERIFY( layer->isValid() );
QCOMPARE( layer->crs().authid(), QStringLiteral( "EPSG:28356" ) );
@ -174,6 +189,27 @@ void TestQgsPdalProvider::validLayer()
QCOMPARE( layer->dataProvider()->pointCount(), 253 );
QCOMPARE( layer->pointCount(), 253 );
QVERIFY( layer->dataProvider()->indexingState() == QgsPointCloudDataProvider::NotIndexed );
}
void TestQgsPdalProvider::testEptGeneration()
{
QTemporaryDir dir;
QVERIFY( dir.isValid() );
QgsPdalEptGenerationTask task( mTestDataDir + QStringLiteral( "point_clouds/las/cloud.las" ), dir.path() );
QVERIFY( task.run() );
QFileInfo fi( dir.path() + "/ept.json" );
QVERIFY( fi.exists() );
}
void TestQgsPdalProvider::testEptGenerationNonASCII()
{
QTemporaryDir dir;
QVERIFY( dir.isValid() );
QgsPdalEptGenerationTask task( mTestDataDir + QStringLiteral( "point_clouds/las/cloud%!* _*čopy.las" ), dir.path() );
QVERIFY( task.run() );
QFileInfo fi( dir.path() + "/ept.json" );
QVERIFY( fi.exists() );
}
QGSTEST_MAIN( TestQgsPdalProvider )

BIN
tests/testdata/point_clouds/las/cloud%!* _*čopy.las vendored Normal file
View File

Binary file not shown.