avinal/avinal.github.io
1
0
Fork 0
mirror of https://github.com/avinal/avinal.github.io.git synced 2026-07-03 23:30:09 +05:30
Code Packages Releases Activity

add gsoc posts

Browse Source 
Signed-off-by: Avinal Kumar <avinal.xlvii@gmail.com>
This commit is contained in:
Morumotto
2023-01-30 22:21:36 +05:30
parent 6ddc56d3a7
commit eae6b0d3f7
15 changed files with 2015 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
static/content/posts/development/cmake-basics.md
+311
View File
@@ -0,0 +1,311 @@
---
title: Hello CMake
date: 2021-05-24 23:56
slug: cmake-basics
category: development
tags: [cmake, gsoc, fossology, gsoc21]
description: "CMake stands for Cross-platform Make. Normally, a build tool like Make
will parse a configuration file (Makefile) that contains all the
commands required to build an artifact based on the source files and
other resources inside the project."
image: "/images/cmake-office.webp"
---
# Hello CMake
CMake stands for Cross-platform Make. Normally, a build tool like Make
will parse a configuration file (Makefile) that contains all the
commands required to build an artifact based on the source files and
other resources inside the project.
I proposed a new architecture for FOSSology that uses CMake instead of
bare-metal Make as a Google Summer of Code 2021 project. Although these
tutorials will be useful for anyone interested in learning CMake they
are specifically tailored to the FOSSology project. This is the first
blog on CMake in this series. In this blog, I will discuss CMake syntax
and various features.
## What is CMake?
![You CMake me happy](/images/cmake-happy.webp)
CMake stands for Cross-platform Make. Normally, a build tool like Make
will parse a configuration file (Makefile) that contains all the
commands required to build an artifact based on the source files and
other resources inside the project. On the other hand, CMake will also
parse a configuration file (CMakeLists.txt), but instead of directly
build the artifact, it’ll generate another configuration file that will
build the artifact.
This approach is very common in Computer Science and is called *adding
another level of indirection*. In short, you may say that:
With just one configuration file you’ll be able to generate different configuration files to build your project for different platforms (Make, Visual Studio, etc).
Another nice CMake feature is the so-called out-of-source build. Any
file required for the final build, executables included, will be stored
in a separated build directory (usually called build/). This prevents
cluttering up the source directory and makes it easy to start over
again: just remove the build directory and you are done.
## CMake Syntax
![Compiling](https://imgs.xkcd.com/comics/compiling.png)
CMake unlike Make is a configuration language itself. CMake supplies a
rich library of inherent functions as well as common programming
language features like conditions, looping, macros, and functions. In
addition to that CMake is highly modular and you can always write a
CMake module yourself independent of any project. Specifically for C/C++
programming, it supplies commands to find and link libraries
automatically and lot many features.
### Language Rules
As mentioned above CMake is a language itself hence there are some
language rules related to syntax, comments, variables, etc.
- There are two types of comment in CMake, both start with `#`
character. The first one is line comments, as clear by name it is
delimited by a newline. The other one is bracket comment and can
span until the matching brackets are found.
```cmake
# This is a line comment and it ends with the line.
#[[This is a bracket comment and it can span up to multiple lines.
But it is only supported in CMake 3.0 or later.]]
```
- Variables in CMake are like any other programming language. They are
case-sensitive and have any alphanumeric characters. In general, it
is recommended using upper case names as variables. They can be
assigned and unassigned using `set` and `unset` commands. A variable
can be referenced using `${VARIABLE_NAME}`.
> CMake reserves some types of identifers:
>
> - begin with **CMAKE_**(upper-, lower-, or mixed-case)
> - begin with ***CMAKE***(upper-, lower-, or mixed-case)
> - begin with **_** followed by the name of any CMake Command
- The CMake commands are case insensitive in the latest version (3.0)
of CMake. That means `message()`, `Message()` or `MESSAGE()` are all
same.
### Basic CMake Commands
- The `project()` command is used to set the name of the CMake project
and optionally a language that is used by the project. Every
top-level CMake configuration must have this option set. The syntax
is as below:
```cmake
project (projectname [CXX] [C] [Java] [NONE])
```
If no language is specified then CMake defaults to supporting C/C++.
If `NONE` is specified then no language support is enabled.
- The `set()` command is used to set a variable to a value or lists of
values. It is one of the most used CMake commands. The accompanying
command is `unset()`. The `unset()` command is used to unset a
variable or remove a variable from the current scope. The syntax for
the three commands are:
```cmake
set (BLOG_TITLE "CMake Introduction") # assign single value
set (BLOG_TAGS "gsoc" "cmake" "fossology" "gsoc21") # assign a list of values
unset (BLOG_TITLE) # unset BLOG_TITLE
```
- The `message()` command can be used to display formatted messages
with different alert modes. There are many
[modes](https://cmake.org/cmake/help/v3.20/command/message.html#general-messages)
of displaying messages. The syntax is :
```cmake
message ([<mode>] "message text" ...)
message(NOTICE "Hey this is ${BLOG_TITLE}") # Example message with variable
```
- The `cmake_minimum_required()` is used to set the minimum CMake
version to use to generate the build files. If any older version is
used than specified then the user gets an error message. It must be
specified at the top of the *CMakeLists.txt* file.
```cmake
cmake_minimum_required (VERSION 3.0)
```
- The commands `add_executable()` and `add_library()` specifies what
executables and libraries to build and what source files must be
used to build them. One of the two commands must be used for any
binary generation.
```cmake
add_executable(<name> [WIN32] [MACOSX_BUNDLE]
[EXCLUDE_FROM_ALL]
[source1] [source2 ...])
add_library(<name> [STATIC | SHARED | MODULE]
[EXCLUDE_FROM_ALL]
[<source>...])
```
### Flow Control
CMake provides three flow control structures. They are conditional
statements (`if`), looping constructs (`foreach` and `while`) and
procedure definitions (`function` and `macro`). I will explain each of
them one by one.
- **Conditional Statements** The `if` command in CMake is just like
the `if` command in any other language. It evaluates its expression
and based on that either executes the code in its body or optionally
the code in the `else` clause.
```cmake
if (FOO)
# do something here
elseif (BAR)
if (NESTED_BAR)
# do something nested here
endif(NESTED_BAR)
# do something else
else ()
# do something here
endif (FOO)
```
You can use many operators to form complex conditions. Available
options are **NOT**, **AND**, **OR**, **COMMAND**, **DEFINED**,
**EXISTS**, **IS_DIRECTORY**, **IS_ABSOLUTE**, **MATCHES**,
**IS_NEWER_THAN**, and operators for numerical comparisons
**EQUAL**, **LESS**, **GREATER**, **STRLESS**, **STREQUAL**,
**STRGREATER**.
- **Looping Constructs** The `foreach` command enables you to execute
a group of CMake commands repeatedly on the members of a list.The
first argument of the foreach command is the name of the variable
that will take on a different value with each iteration of the loop.
The remaining arguments are the list of values over which to loop.
```cmake
foreach(<loop_var> <items>)
<commands>
endforeach()
```
The `while` command provides for looping based on a test condition.
The format for the test expression in the `while` command is the
same as that for the `if` command described earlier.
```cmake
while(<condition>)
<commands>
endwhile()
```
It is worth mentioning that foreach loops can be nested and that the
loop variable is replaced prior to any other variable expansion.
This means that in the body of a foreach loop you can construct
variable names using the loop variable.
- **Procedure Definitions** A function in CMake is very much like a
function in C or C++. You can pass arguments into it, and the
arguments passed in become variables within the function. The first
argument is the name of the function to define. All additional
arguments are formal parameters to the function.
```cmake
function(<name> [<arg1> ...])
# write the function body here
<commands>
endfunction()
```
Macros are defined and called in the same manner as functions. The
main differences are that a macro does not push and pop a new
variable scope, and the arguments to a macro are not treated as
variables but are string replaced prior to execution. This is very
much like the differences between a macro and a function in C or
C++. The first argument is the name of the macro to create. All
additional arguments are formal parameters to the macro.
```cmake
macro(<name> [<arg1> ...])
# write macro definition here
<commands>
endmacro()
```
## A Hello CMake example
This example compiles a simple *hello_cmake* program. This example and
the terminal commands are used in Linux context, however there is very
little difference in different platforms. Make sure to [install
CMake](https://cmake.org/install/) for your platform.
- Create a folder and create a file named `hello_cmake.cpp` in that.
You may add your own code. Here is my example code.
```cpp
#include<iostream>
int main() {
std::cout << "Hello CMake\n";
return 0;
}
```
- Create another file named `CMakeLists.txt` and add the following
script in that file.
```cmake
cmake_minimum_required(VERSION 3.0)
# set project name
project(hello_cmake)
# print compiler info
message("The compiler is ${CMAKE_CXX_COMPILER}")
# add executable
add_executable(Hello_cmake hello_cmake.cpp)
```
- Create another directory `build` and run the following commands.
```bash
# create folder and change directory
mkdir build && cd build
# run cmake config
cmake ..
# build project
cmake --build .
```
You will be able to see a `Hello_cmake` binary in the *build* folder.
Hooray you have successfully built a project using CMake. For more [read
here](https://cmake.org/cmake/help/v3.20/guide/tutorial/index.html). In
the next blog I will explain how to create CMake configuration for a
more complex project.
Thanks!
## References and Credits
- [CMake Website](https://cmake.org)
- [CMake
Documentation](https://cmake.org/cmake/help/latest/index.html)
- [Mastering CMake Book](https://www.kitware.com/what-we-offer/#books)
- [CMake
Tutorial](https://cmake.org/cmake/help/v3.20/guide/tutorial/index.html)
- [You (C)Make Me
Happy](https://prateekvjoshi.com/2014/02/01/cmake-vs-make/)
- [Compiling xkcd.com](https://xkcd.com/303/)
static/content/posts/gsoc/final-evaluation.md
+831
View File
@@ -0,0 +1,831 @@
---
title: GSoC'21 Final Evaluation Report
date: 2021-08-19 23:07
tags: [gsoc, FOSSology]
category: gsoc
description: This is the final report of my Google Summer of Code 2021 at The FOSSology Project.
image: "/images/gsoc-wall.webp"
---
# GSoC'21 Final Evaluation Report
<style>
.rd {color:red;font-weight:bold}
.gr{color:green;font-weight:bold}
.or{color:orange;font-weight:medium}
ul{margin-bottom:0}
</style>
## The CMake Build system
FOSSology is quite an old and mature project. The project has been using
bare metal **Makefiles**. As the project is growing with new agents and
modernization it was required to have a modern build system.
The FOSSology is a suite of well-integrated and synchronized
sub-projects (called agents) written in C, C++, and PHP. Most of the
major agents are in C, C++ and that made CMake an obvious choice for a
new build system for FOSSology. CMake is a versatile set of build, test,
and packaging tools and is the most popular choice of C/C++ developers.
CMake can be extended to create a build system for other languages too
via custom scripts.
## GitHub Actions CI/CD
<img src="/images/ci.webp"
class="float-md-right rounded border border-info ml-3 float-md-right rounded border border-info ml-3"
width="350" alt="A CI Meme" />
Since the FOSSology project moved on Github, it has used only the free
Travis CI service for OSS projects. At the time of writing Travis CI has
reduced its free tier CI services. GitHub Actions provides faster
builds. Since GitHub Actions is a fully managed service by GitHub, we
don't need to know how to scale and operate the infrastructure to run
it.
It is straightforward to use with GitHub because when we fork a
repository, the actions automatically get forked. This allows you to
test and build projects very efficiently and even run them closer to the
developer. Also, you have readily available access to the GitHub API,
making it more popular among developers.
## Improvements over previous build system and CI
The new build system and CI brings a lot of improvements and features.
The list below describes them.
- CMake enforces out-of-source builds. This was already possible with
the previous build system but not a strict requirement. This feature
keeps the source code clean and makes cleaning the build artifacts
easy. (Just remove the build folder :)
- One of the major improvements over the previous build system is faster
build times. CMake generates parallel build-enabled configurations for
all generators. In our tests, the new build system is at least twice
as fast as the previous one. With further improvement in
configuration, we will be able to further cut the build times.
- Previously FOSSology can only be built using *Unix Makefiles*. With
CMake, we can now use many other popular generators such as *Ninja*.
- Now it is also very flexible to choose different compilers. This will
help support more platforms and architecture in the future. As of now,
we are experimenting with Clang compilers.
- FOSSology is quite an old project and a lot of agent testing code was
written in the last decade. Initially, none of them were compatible
with the new build system, but we were able to hack most of the test
code using better-improved methods. Test times have also improved.
- Migrating from Travis CI to GitHub Actions was another big move and
for the most part, it removes the dependency on a third-party CI
service. Along with that GitHub Actions provides better build times,
tons of new features, and better integration with other GitHub
services.
## Deliverables
- Final Pull Request [#2075](https://github.com/fossology/fossology/pull/2075)
- Pull Request Branch [avinal/feat/buildsystem](https://github.com/avinal/fossology/tree/avinal/feat/buildsystem)
- Working Branch (individual commits)
- [avinal/feat/cmake-buildsystem](https://github.com/avinal/fossology/tree/avinal/feat/cmake-buildsystem)
- [avinal/feat/testing](https://github.com/avinal/fossology/tree/avinal/feat/testing)
- Project Issue [#1913](https://github.com/fossology/fossology/issues/1913)
- Project Discussion [#1931](https://github.com/fossology/fossology/discussions/1931)
- Weekly Reports
- [Personal Blog](https://gsoc.avinal.space)
- [FOSSology Official Blog](https://fossology.github.io/gsoc/docs/2021/buildsystem/)
### CMake Build System Tasks
<table>
<colgroup>
<col style="width: 7%" />
<col style="width: 23%" />
<col style="width: 15%" />
<col style="width: 15%" />
<col style="width: 30%" />
<col style="width: 23%" />
<col style="width: 38%" />
</colgroup>
<thead>
<tr class="header">
<th>#</th>
<th>Agents</th>
<th>Build</th>
<th>Install</th>
<th>Testing</th>
<th>Packaging</th>
<th>Remarks</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>1</td>
<td>adj2nest</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>2</td>
<td>buckets</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><blockquote>
<p><span class="gr">YES</span></p>
</blockquote></td>
<td></td>
</tr>
<tr class="odd">
<td>3</td>
<td>cli</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>4</td>
<td>copyright</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>5</td>
<td>debug</td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>6</td>
<td>decider</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>7</td>
<td>deciderjob</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>8</td>
<td>delagent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">Functional</span></li>
<li><span class="rd">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>9</td>
<td>demomod</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td><ul>
<li><span class="or">Functional</span></li>
<li><span class="or">Unit</span></li>
</ul></td>
<td><span class="or">NO</span></td>
<td><em>(Not Used)</em></td>
</tr>
<tr class="even">
<td>10</td>
<td>example_wc_agent</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td><ul>
<li><span class="or">Functional</span></li>
<li><span class="or">Unit</span></li>
</ul></td>
<td><blockquote>
<p><span class="or">NO</span></p>
</blockquote></td>
<td><em>(Not Used)</em></td>
</tr>
<tr class="odd">
<td>11</td>
<td>clib</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>12</td>
<td>cpplib</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>13</td>
<td>phplib</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>1 functional test needs fix</td>
</tr>
<tr class="even">
<td>14</td>
<td>maintagent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>15</td>
<td>mimetype</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>16</td>
<td>monk</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>17</td>
<td>ninka</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td><ul>
<li><span class="or">Functional</span></li>
<li><span class="or">Unit</span></li>
</ul></td>
<td><span class="or">NO</span></td>
<td><em>(Deprecated)</em></td>
</tr>
<tr class="even">
<td>18</td>
<td>nomos</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>19</td>
<td>ojo</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>1 functional test needs fix</td>
</tr>
<tr class="even">
<td>20</td>
<td>pkgagent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>21</td>
<td>readmeoss</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>22</td>
<td>regexscan</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td></td>
<td><blockquote>
<p><span class="or">NO</span></p>
</blockquote></td>
<td><em>(Deprecated)</em></td>
</tr>
<tr class="odd">
<td>23</td>
<td>reportImport</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>24</td>
<td>reuser</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>25</td>
<td>reso</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>26</td>
<td>scheduler</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">Functional</span></li>
<li><span class="rd">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>Tests needs fix</td>
</tr>
<tr class="odd">
<td>27</td>
<td>softwareHeritage</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>28</td>
<td>spasht</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>29</td>
<td>spdx2</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>1 Test failing in CI</td>
</tr>
<tr class="even">
<td>30</td>
<td>unifiedreport</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>31</td>
<td>ununpack</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="rd">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>Unit tests needs fix</td>
</tr>
<tr class="even">
<td>32</td>
<td>wget_agent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>32</td>
<td>www</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">UI</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
</tbody>
</table>
### GitHub Actions CI Tasks
| # | CI Tasks | Status |
|-----|-----------------------------------------|----------------------------------------------------------|
| 1 | <span class="gr">build</span> | Added Ubuntu 20.04 GCC 8, 9 and Clang, GCC 7 not working |
| 2 | <span class="gr">c/cpp unit test</span> | Added, delagent, scheduler and ununpack not working |
| 3 | <span class="gr">phpunit tests</span> | Added, delagent and scheduler functional not working |
| 4 | <span class="rd">cahching</span> | Not implemented |
| 5 | <span class="rd">source install</span> | Not implemented |
(<span class="gr">GREEN</span>: COMPLETED, <span class="rd">RED</span>:
INCOMPLETE, <span class="or">ORANGE</span>: NOT NEEDED/DEPRECATED)
## How does it work and how to use it?
The new build system retains the modular and hierarchical structure of
the previous build system. On the other hand, the new build system
provides several new flags to control the build. The new build system
forces out-of-source build instead of the previous in-source builds.
This keeps the source clutter-free and reduces the chance of
accidentally deleting source files. *Testing still needs some in-source
artifacts, this will be solved once all the tests are fixed according to
the new build system.*
Each agent is a complete CMake sub-project with its independent
configuration for building, installing, and testing. That means a single
agent can be built and installed separately and even removed from the
default build without breaking other builds. The directory structure is
as below.
```bash
.
├── build # temporary directory for build artifacts
├── cmake # CMake modules for FOSSology
│ ├── FoPackaging.cmake # CMake Packaging configurations
│ ├── FoUtilities.cmake # Custom CMake utilities
│ ├── FoVersionFile.cmake # VERSION version.php CMake template file
│ ├── SetDefaults.cmake # CMake defaults for this project
│ ├── TestInstall.make.in # Template makefile for install during tests
│ └── VERSION.in # VERSION file template
├── src
│ ├── agent-1 # Agent sub-project
│ │ ├── agent # Agent's source code directory
│ │ │ ├── agent-source-code
│ │ │ └── CMakeLists.txt
│ │ ├── agent_tests # Agent's test directory
│ │ │ ├── Unit
│ │ │ ├── Functional
│ │ │ └── CMakeLists.txt
│ │ ├── ui # Agent's UI source code
│ │ │ ├── templates
│ │ │ └── agent-ui-code
│ │ └── CMakeLists.txt # Agent's top-level CMake configuration
: :
│ ├── other agents
: :
│ └── CMakeLists.txt # Source intermediate CMake configuration
:
├── other directories and files
:
└── CMakeLists.txt # FOSSology Top-level CMake configuration
```
The `cmake` directory contains customized CMake modules and templates
for FOSSology. This directory is required for all the operations. The
general workflow of the new build system as well as how to use it is
described below.
1. Since the new build system is still in review. You must fork
FOSSology and pull the
[#2075](https://github.com/fossology/fossology/pull/2075) pull
request branch. Once you are in FOSSology root, run these commands.
```bash
git fetch https://github.com/avinal/fossology avinal/feat/buildsystem:buildsystem
git checkout buildsystem
```
2. The first step towards building is to create a temporary directory
for storing intermediate files and build artifacts. By convention we
use a directory named `build`, but you can use any name. (**NOTE:
For testing do not use other names**)
```bash
mkdir build
cd build
```
3. In the next steps, we will configure the CMake project and generate
the required configurations. You can use several flags to control
the build. Given below are the flags available for this project.
<table style="width:99%;">
<colgroup>
<col style="width: 34%" />
<col style="width: 43%" />
<col style="width: 20%" />
</colgroup>
<thead>
<tr class="header">
<th>CMake Flags</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>-DCMAKE_INSTALL_PREFIX=&lt;path&gt;</strong></td>
<td>Sets the install prefix.</td>
<td><code>/usr/local</code></td>
</tr>
<tr class="even">
<td><strong>-DAGENTS="agent1;agent2..."</strong></td>
<td>Only configure these agents.</td>
<td>ALL AGENTS</td>
</tr>
<tr class="odd">
<td><strong>-DOFFLINE=&lt;ON/OFF&gt;</strong></td>
<td>Controls vendor generation, ON=NO</td>
<td><strong>OFF</strong></td>
</tr>
<tr class="even">
<td><p><strong>-DCMAKE_BUILD_TYPE=&lt;type&gt;</strong></p>
<blockquote>
<ul>
<li>Controls build type aka level optimisation</li>
</ul>
</blockquote></td>
<td><ul>
<li><code>Debug</code></li>
<li><code>Release</code></li>
<li><code>RelWithDebInfo</code></li>
<li><code>MinSizeRel</code></li>
</ul></td>
<td><code>Debug</code></td>
</tr>
<tr class="odd">
<td><strong>-DTESTING=&lt;ON/OFF&gt;</strong></td>
<td>Controls testing config generation</td>
<td><blockquote>
<p><strong>OFF</strong></p>
</blockquote></td>
</tr>
<tr class="even">
<td><strong>-DMONOPACK=&lt;ON/OFF&gt;</strong></td>
<td>Package adj2nest and ununpack seperately</td>
<td><strong>OFF</strong></td>
</tr>
<tr class="odd">
<td><strong>-GNinja</strong></td>
<td>Use Ninja instead of Unix Makefiles</td>
<td><em>Unix MakeFiles</em></td>
</tr>
</tbody>
</table>
There are lots of inbuilt CMake command-line options you can see
them in the official
[documentation](https://cmake.org/cmake/help/v3.10/manual/cmake.1.html).
Once you have chosen your flags we can now configure the project
using the following commands.
```bash
# From build folder
cd <name-of-build-directory>
cmake <flags> ..
```
4. The next step is to build the project. You can use parallel jobs to
build faster. For more options you can type `cmake --help` or
`make --help` or `ninja --help`.
```bash
# Common build command for all generators,
# Default number of parallel builds depends on generator used
cmake --build . --parallel <no-of-processes>
# For Unix Makefiles, no parallel build by default
make -j <no-of-processes>
# For Ninja, 8+ parallel build by default (depends on system)
ninja -j <no-of-processes>
```
5. Installing is also as easy as building. You can choose to install
only certain components even if you have built the whole project. If
you directly invoke the install command without building the
project, it will automatically build the project first.
```bash
# For Unix Makefiles
make install
# For Ninja
ninja install
```
6. While testing has some issues, most of the testing is working fine.
For now, you must build and run any test from the FOSSology root
directory only. You can choose to configure a single agent if you
want to test one agent only. See `ctest --help` for controlling test
runs.
```bash
# Common testing command
ctest --parallel <no-of-processes>
# For Unix Makefiles
make test
# For Ninja
ninja test
```
7. You can package FOSSology, the packaging currently lacks copyright
and conf files. But for testing purposes, you can use the following
commands. Similar to installing, if you run the package command
without building the project, it will automatically build the
project first. See `cpack --help` for more packaging options.
```bash
# Common testing command
cpack
# For Unix Makefiles
make package
# For Ninja
ninja package
```
## Known Issues and Drawbacks
Although the transition from Makefiles to CMake and Travis CI to GitHub
Actions is almost complete and working as expected. But it is not free
of drawbacks and issues. This section outlines the known issues at the
time of writing.
![A Bug Meme](https://imgs.xkcd.com/comics/conference_question.png)
- Coverage builds may fail with linking errors.
- Packaging prefix is the same as the install prefix. This requires the
developer to set the install prefix manually before packaging to
produce packages with the correct directory structure.
- Testing and packaging must be used from the FOSSology root directory.
Not doing so may or may not configure the project as intended.
- Previously tests were written hardcoded for the Makefiles. But new
build system requires all artifacts to be generated in a separate
directory. This required me to add symbolic links wherever a generated
script or file is expected. Tests can still leave some artifacts
inside source folders.
- There is no easy way to install a particular agent from the FOSSology
root directory.
- Packages don't contain copyright, readme, and license files. CMake
doesn't provide a way to include these files. This is being tracked by
issue
[#21832](https://gitlab.kitware.com/cmake/cmake/-/issues/21832).
- While packaging the symbolic links may or may not be dereferenced and
hence results in copying the folder too in the target directory.
- Running tests locally may require switching to `fossy` user.
- While configured for testing, it may give permission errors.
- Scheduler, Ununpack, and Delagent unit and functional tests are not
working. I have added an issue
[#2084](https://github.com/fossology/fossology/issues/2084) to track
the progress on fixing these tests.
- CMake doesn't generate uninstall targets. The closest thing to
uninstall is [this snippet](https://gitlab.kitware.com/cmake/community/-/wikis/FAQ#can-i-do-make-uninstall-with-cmake).
This will be later added to the FOSSology.
## Challenges Faced
While this whole project was challenging, some aspects of it were
unforeseen and more challenging. When I decided to go on with this
project I just had enough CMake knowledge to write a configuration for a
very small project. I had never used CMake on this big scale. On the
other side, the FOSSology community is largely unknown to CMake so for
all of us it was learned, practiced, and implement. With support from
mentors, I was able to overcome this challenge with flying colors.
The other challenge was to understand the old build system, how they are
all connected and what is the flow. The complexity can be imagined by
the fact that the most of code and configurations were written in the
decade before the last decade and haven't changed much since then.
The most challenging task was to make tests work with the new build
system. Since tests were mostly hardcoded and the new build system
refactored many of the files and directory, the tests were failing
initially. The testing part took me the most time. All thanks to my
mentor Gaurav, I was able to hack them to suit the
new build system.
## Related Resources and Links
- Fix FOSSology agent tests issue
[#2084](https://github.com/fossology/fossology/issues/2084)
- feat(CI): Migrate API docs generation and deployment to GitHub Actions
pull request
[#1917](https://github.com/fossology/fossology/pull/1917)
- feat(CI): Migrate Static Checks and Analysis to GitHub Actions from
Travis CI [#1919](https://github.com/fossology/fossology/pull/1919)
## Future Development Plans
There is a lot to do with the new build system and CI and it will
probably take a year or to reach a maturity point. I was able to meet
most of the goals but some of them are remaining.
- Fix the tests, probably renovate them from the ground up.
- Find a hack for packaging problems.
- Improve and optimize the build.
- Modernise the source code, remove old, bloated code and replace them
according to new standards.
## What did I learn from this project?
This Google Summer of Code was the busiest time of my life for all good
reasons. I learned a lot about license compliance and how it all works
in the software industry. The next big thing is CMake. As I mentioned I
was just a novice user of CMake. Now I am confident that given any other
large project I will be able to migrate it/improve it. I got to learn
PHP, of which I did not know a single word before GSoC. And finally, I
learned about packing and testing. I had these courses but implementing
them myself and fixing them was a wholesome experience.
Other than that I improved on my communication and presentation skills.
Collaborating with fellow participants was one of the great things that
happened during GSoC.
## Acknowledgments
Google Summer of Code is the best thing that has happened to me this
year so far. Although there are numerous people to say thanks to, I want
to mention key people who were my motivation and support during this
period.
First of all, I want to thank and appreciate my mentors [Gaurav
Mishra](https://github.com/GMishx), [Michael C.
Jaeger](https://github.com/mcjaeger), [Anupam
Ghosh](https://github.com/ag4ums), and [Shaheem Azmal M
MD](https://github.com/shaheemazmalmmd). Without the help and support
from them, all this would not have been possible. They are very polite,
knowledgeable, and helpful.
Finally, I want to thank, my family and friends. I got to meet many
awesome developers as my fellow participants from around the world, I
wish we will do more collaboration in the future.
static/content/posts/gsoc/first-evaluation.md
+72
View File
@@ -0,0 +1,72 @@
---
title: GSoC'21 First Evaluation Report
date: 2021-07-14 12:29
tags: [gsoc, FOSSology]
category: gsoc
description: "In the first phase of GSoC 2021 @ The FOSSology Project, I have
completed the desired milestone. As of now, FOSSology can be installed
completely via CMake and most of the components are working fine in
initial testing."
image: "/images/tech-wallpaper.webp"
---
# GSoC'21 First Evaluation Report
In the first phase of GSoC 2021 @ The FOSSology Project, I have
completed the desired milestone. As of now, FOSSology can be installed
completely via CMake and most of the components are working fine in
initial testing.
## Updates
In the first phase of GSoC 2021 @ The FOSSology Project, I have
completed the desired milestone. As of now, FOSSology can be installed
completely via CMake and most of the components are working fine in
initial testing.
List of tasks completed
- Added CMake build configurations for all the C/C++ agents for
executables, libraries, and coverages
- Added CMake install configuration for all C/C++ and PHP agents as well
as extra components
- Reworked the shell scripts and generated source files to make them
more compatible with CMake as well as better in terms of overall
compatibility with the latest tools.
## Improvements
- The new CMake build architecture is much more flexible to changes as
compared to hard-coded Makefiles.
- CMake generated configurations support parallel build by default, this
has led to significant improvement in build time. CMake generated
configuration can now build the whole project within 2 mins or even
faster on more powerful CPUs (Both Ninja and Makefiles with the same
number of parallel processes) compared to 4-5 minutes previously.
*(These results are averaged from initial testing of new build
architecture)*
- CMake supports out-source builds by default, which means the source
folders are not touched/modified while building, all build files and
residuals get their separate folder and the source tree can be cleaned
easily.
- Developers can now opt for a long list of generators to build
FOSSology e.g Makefiles, Ninja as per their needs.
## How to test
Instructions to test the new Build system is in [this](https://github.com/avinal/fossology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now) wiki.
## Known Issues
- There may be a permission issue with some generated sources while
building. This can be bypassed for now by running
`sudo chmod +x <filename>` command.
- Coverage builds may fail.
## Postponed Tasks
- configuration for tests are skipped for now
## Work in Progress
- Currently, I am working on packaging the FOSSology with CMake.
static/content/posts/gsoc/meeting-0.md
+49
View File
@@ -0,0 +1,49 @@
---
title: Community Bonding Meeting 0
date: 2021-05-28 21:00
tags: [gsoc, fossology]
category: gsoc
description: "This meeting is the first of the recurring weekly GSoC project meetings.
In this meeting the current status of progress according to the proposal
was discussed and some topics related to current build system based on
Make and the new build system based on CMake."
image: "/images/tech-wallpaper-1.webp"
---
# Community Bonding Meeting 0
This meeting is the first of the recurring weekly GSoC project meetings. In this meeting the current status of progress according to the proposal was discussed and some topics related to current build system based on Make and the new build system based on CMake.
## Discussions
- **The current progress according to schedule**
- The blog on CMake is on the way.
- I have gone through the Makefiles to get a rough estimate of the
work.
- Published the GSoC project blog
- **How are agents related to each other in terms of compilation?**
- Each agent is independently compiled and generally use the source
code in `lib` folder. If any agent needs other agent then it uses
the library files instead.
- **Does every agent have a executable and library?**
- Not necessarily, there are agents written in C, C++ and PHP,
depending on what is the use the configuration can be different.
## Conclusion and Further Plans
- It would be better if I get started by creating CMake configuration
for any of the agent.
- Fork and create a branch for development and mention the same in blog
or wiki.
- Add a timeline section in blog or wiki as provided in the project
proposal.
- Publish the CMake introductory blog.
- Prepare a prototype/plan for next week.
- Find out the best alternative for handling the global variables.
## Attendees
- [Gaurav Mishra](https://github.com/GMishx)
- [Anupam Ghosh](https://github.com/ag4ums)
- [Ayush Bhardwaj](https://github.com/hastagAB)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-1.md
+41
View File
@@ -0,0 +1,41 @@
---
title: Community Bonding Meeting 1
date: 2021-06-04 22:30
tags: [gsoc, fossology]
category: gsoc
description: "In this second meeting points over default Makefiles were discussed.
Ninja can be used as an alternative for Makefiles."
image: "/images/tech-wallpaper-2.webp"
---
# Community Bonding Meeting 1
In this second meeting points over default Makefiles were discussed. Ninja can be used as an alternative for Makefiles.
## Discussions
- **What is the use of** `Makefile.deps` **and** `Makefile.process`
**files?**
- `Makefile.deps` consists of many used and unused snippets. These
snippets help setup the build and test environment for fossology.
Since there are many directories that are hardcoded, special care is
required while replacing this file.
- `Makefile.process` generates a master variable from list of
variables. It assists the script in `Makefile.conf` file. These
files together generate a list of variables that can be used
throughout the build process.
- The build can be made faster using **Ninja** instead of **Make**.
- Ninja supports parallel builds by default.
- Print the flags used once the CMake configuration is working. That
will help us debug the process.
## Conclusion and Further Plans
- Write a *CMakeLists.txt* for **lib**.
- Push the working branch and update the link either on wiki or blog.
## Attendees
- [Gaurav Mishra](https://github.com/GMishx)
- [Anupam Ghosh](https://github.com/ag4ums)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-10.md
+100
View File
@@ -0,0 +1,100 @@
---
title: Coding Week 9 Meeting
date: 2021-08-06 22:47
tags: [gsoc, FOSSology]
category: gsoc
description: "This week I worked on CMake testing configuration. Most of the time was
spent understanding the previous testing architecture."
image: "/images/tech-wallpaper-11.webp"
---
# Coding Week 9 Meeting
This week I worked on CMake testing configuration. Most of the time was spent understanding the previous testing architecture.
## Week 9 Progress
> Initial CMake testing configuration added.
>
> - Few tests working, e.g copyright, nomos
> - Improved packaging configurations
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **Is this a necessity that tests must be run as the fossy user?
Because when I run tests as me they as for permissions. But proceeds
as the fossy user.**
- No this is not required and this should not happen. They run under
fossy as they sometimes require writing into /srv/fossology. But if
they can run under other users that is an enhancement.
- **I am getting a lot of install issues in C/C++ agent tests?**
``` bash
Start 3: delagent_unit_test
3: Test command: /home/avinal/Documents/my_git/fossology/build/src/delagent/agent_tests/test_delagent
3: Test timeout computed to be: 10000000
3: install: cannot stat '/home/avinal/Documents/my_git/fossology/build/src/delagent/agent_tests/..//../../install/defconf/Db.conf': No such file or directory
3: install: cannot stat '/home/avinal/Documents/my_git/fossology/build/src/delagent/agent_tests/..//VERSION': No such file or directory
3: sh: 1: ../../../testing/db/createTestDB.php: not found
3: Failed to run ../../../testing/db/createTestDB.php -c /home/avinal/Documents/my_git/fossologbuild/src/delagent/agent_tests/testconf -e, exit code is:127 .
3/8 Test #3: delagent_unit_test ...............***Failed 0.02 sec
```
- Not sure about the reason. I was suspecting Makefile but since they
are gone now, I think PHP files are calling some shell commands
causing this.
- **Suggestions/Changes from Gaurav for fixing tests.**
- For clib-tests, it needs to be called from PHP file (via PHPUnit) as
it requires setting up a dummy repo. Check the
`src/lib/c/test/Makefile`
- For missing services.xml, the test cases include
`src/lib/php/common-container.php` which loads the file. It expects
it to be in current dir. Can be solved in two ways
- Create another common-container.php just for test cases with
correct paths.
- Edit the current file and take the help of environment variables.
For example, if a test variable is exported in env, find the XML
relative to it otherwise continue as normal and this variable can
be exported by CMake during the test.
- Scheduler tests do need `fossology_testconfig` from Makefile.deps
which set up the srv and create test configurations, DB, etc.
- Another shell script can be written to do all that and call it
from CMake. The PHP file called makes everything required in /tmp
so not an issue.
- The locations like `LOG_DIR, FOSSDB_CONF`, etc. in CMakeLists.txt
can be changed to some other values. I am guessing this is the
reason you were asked for the fossy password.
- File `src/copyright/agent_tests/Functional/cli_test.sh` needs to be
edited to take paths relative to build dir. It can also be made into
a .in file which is generated from CMake? So every path can easily
be updated.
- For PHP agents with missing version.php issue, there is a hack
possible
- Check
<https://www.php.net/manual/en/function.set-include-path.php>
- Another hack will be to use soft links for version.php in the
source.
- Other PHP issues like
`PHP Fatal error: Uncaught Error: Class 'Fossology\Lib\Agent\Agent' not found`
can only be solved by editing composer.json before doing composer
install (look for autoload: psr-4 ).
- For delagent, pkgagent, mimetype issues, something can be done here:
<https://github.com/avinal/fossology/blob/avinal/feat/testing/src/testing/db/c/libfodbreposysconf.c#L349>
## Conclusion and Further Plans
- Raise a pull request for all the progress till now.
- Refactor the test source code according to suggestions.
- Implement remaining testing configurations.
## Attendees
- [Gaurav Mishra](https://github.com/GMishx)
- [Anupam Ghosh](https://github.com/ag4ums)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-11.md
+79
View File
@@ -0,0 +1,79 @@
---
title: Coding Week 10 Meeting
date: 2021-08-14 22:47
tags: [gsoc, FOSSology]
category: gsoc
description: "This week I implemented CMake testing configuration and fixed most of
the tests. As of now all but 5 tests are working fine."
image: "/images/tech-wallpaper-12.webp"
---
# Coding Week 10 Meeting
This week I implemented CMake testing configuration and fixed most of the tests. As of now all but 5 tests are working fine.
## Week 9 Progress
> Testing configuration for all agents added
>
> - GitHub Actions Configuration added
> - Fixed and refactored most of the tests
> - Raised a pull request for all the works till now. [#2075](https://github.com/fossology/fossology/pull/2075)
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **I suspect that the Ojo regression test's expected data file is
outdated**
- Michael said that on their internal Jenkins CI, these tests are not
being run currently, so this might be possible that the file is
outdated.
- **Since some of the tests need Makefile to install while testing,
CMake generated Makefiles and test Makefiles are conflicting, and
hence we are forced to use Ninja for testing. What can I do about
it?**
- Michael suggested using `--file=filename` flag with the make command
and change the name of the test Makefile to something else. This
will solve the problem.
- **Mimetype is detecting executables as shared lib, is that expected or
needs to be fixed?**
- Mimetype internally depends on the *file* command to get the
mime-type. If the output of the *file* command is also the same then
it is okay.
- **What is** `folderlist` **in
<https://github.com/fossology/fossology/blob/master/src/delagent/agent_tests/Functional/ft_cliDelagentTest.php#L126>
?**
- `folderlist` is a view. Use `createViews()` function.
- **Suggestions/Changes from Gaurav for fixing phpunit tests.**
- Please note the changes in `setUp()` function in
`src/lib/php/tests/test_common_license_file.php`
- The test database name is given to the constructor of TestPgDb and
can be anything as it gets deleted in `teardown()`
- The `dbmanager` is provided by the object, no need to initialize
global `PG_CONN` (it will be exposed by the library in case some of
the functions need it).
- All the tables needs to be explicitly mentioned to
`createPlainTables()` and their corresponding `createSequences()`
(you can get them using `\d tablename` from existing DB easily. Then
call the `alterTables()` to update the sequence. (I am not sure if
`createConstraints()` is required at all, try to remove)
- `tearDown()` is pretty easy, just need to call `fullDestruct()`. For
debugging, you can add `exit(-1);` after any line you as suspecting,
connect to DB and checkout the database, select/inspect tables.
- There is also `TestInstaller` class in case any of test case needs
the whole mods-enabled with fossology.conf, VERSION, etc. Please
check `src/cli/tests/test_fo_copyright_list.php` for quick
reference.
## Conclusion and Further Plans
- Fix the remaining tests.
- Add week 8, 9 reports.
- Add Final Evaluation Report.
- Complete Final Evaluation.
## Attendees
- [Michael C. Jaeger](https://github.com/mcjaeger)
- [Anupam Ghosh](https://github.com/ag4ums)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-2.md
+91
View File
@@ -0,0 +1,91 @@
---
title: Coding Week 1 Meeting
date: 2021-06-11 23:30
tags: [gsoc, FOSSology]
category: gsoc
description: "In this third meeting, I demoed the working build system, currently
building executables and libraries, a lot of queries were resolved about
writing version files and attaching commits and hashes to the build."
image: "/images/tech-wallpaper-3.webp"
---
# Coding Week 1 Meeting
In this third meeting, I demoed the working build system, currently building executables and libraries, a lot of queries were resolved about writing version files and attaching commits and hashes to the build.
## Week 1 Progress
> This week was mainly focused on analyzing the previous build system and framing a skeleton for the new build system.
>
> - Created the build configuration [analysis table](https://github.com/avinal/FOSSology/wiki/agents-spec#agents-configuration-list).
> - Completed the basic skeleton.
> - Completed the CMake configuration for libraries
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
> - Test on [GitPod](https://gitpod.io/#https://github.com/avinal/FOSSology/tree/avinal/feat/cmake-buildsystem) right inside your browser.
## Discussions
- **What are the flags needed for C and C++?**
- The `-g` flag enables debug.
- The `-O2` flag is used for optimizing.
- In FOSSology these two flags are used together by default for all
build purposes because it is desired to have an optimized binary but
some level of debugging information is also desired.
- **The Makefiles have some compile-time preprocessor macro definitions
that need to be passed to each build.** The Makefiles have all the
path values passed as `'"..value.."'` format *(double quote inside
single quotes)*, however the commands produced by CMake have
`\"..value..\"` format *(escaped double quotes)*. Are they the same or
it needs to be changed?
- Currently, there is nothing to determine if they work the same or
not, but if the compiler would not have accepted them then, it would
have thrown an error. As long it is working these should be fine,
but will need to be checked in the final build.
- **Are all libraries in FOSSology static?**
- No, by default no library is static. The format
`lib<library-name>.a` is confusing but no need to worry about it for
now, if this is working fine then no problem.
- In general, this format denotes a static library.
- **How to add the version and commit information to the builds?**
- I have gone through [this
thread](https://cmake.org/pipermail/cmake/2018-October/068383.html)
on CMake's official mailing list. And they have suggested a lot of
options, but unable to decide which option to use. Gaurav said he
will see into this thread and for now, I should try writing a shell
script and test if that works.
- Same can be tested for the version too.
- **What is** `_squareVisitor.h.pre` **used for?**
- They are used to generate source code at build time.
- **Is there any inheritance structure in the build system?**
*(Michael)*
- For now, I am writing separate modules for the default operations
needed in most configurations. The final structure will be decided
in the final build.
- **Where are all the binaries produced?** *(Gaurav)*
- They are located in the build folder with the same directory
structure as the original project.
- While installing the same will be used and none of the source
folders are ever disturbed.
- **Are all flags taken from the Makefiles itself?** *(Anupam)*
- Yes and No, there are some flags that CMake uses by default, they
can be altered by changing the value for `CMAKE_C_FLAGS` and
`CMAKE_CXX_FLAGS`. One can also append their flags. Since not all
compilation requires all the flags, I have taken the default one
into cache variables, and others are appended while configuring for
a particular project.
## Conclusion and Further Plans
- Try the `monkbulk` in monk and `makefile.sa` in nomos.
- Try adding the version and commit hash info.
- Implement writing version files for each build.
- Add proper comments in the `CMakeLists.txt` files.
## Attendees
- [Michael C. Jaeger](https://github.com/mcjaeger)
- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd)
- [Gaurav Mishra](https://github.com/GMishx)
- [Anupam Ghosh](https://github.com/ag4ums)
- [Ayush Bhardwaj](https://github.com/hastagAB)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-3.md
+64
View File
@@ -0,0 +1,64 @@
---
title: Coding Week 2 Meeting
date: 2021-06-18 23:30
tags: [gsoc, FOSSology]
category: gsoc
description: "In this fourth meeting, a lot of questions were discussed related to the
existing build system and what things we have to drop or modify."
image: "/images/tech-wallpaper-4.webp"
---
# Coding Week 2 Meeting
In this fourth meeting, a lot of questions were discussed related to the existing build system and what things we have to drop or modify.
## Week 2 Progress
> This week was mainly focused on creating CMake configuration for libraries, executables and coverage.
>
> - Added the configuration for libraries and executables
> - Resolved parallel build problems with coverage configs
> - Implemented generated source configurations
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **Should I generalize the coverage build for each agent?**
- Coverage depends on the agent_tests and may or may not be available
for all the agent. So follow the Makefiles and add the configuration
as it is in them.
- Leave coverage for them who don't have it already in their
Makefiles.
- **What are :code:\`\$(AGENTLIB) \$(REPO) \$(DB)\` in the Makefiles?**
- They seems to be remains of previous build configuration. Until
there is a problem, ignore if you can not find the definitions.
- **Can I refactor the directory structure of nomos and monk, it will
help keep the source code generation out of source directory?**
- Yeah, sure. As long as it does not affects the working of the
project you may refactor them to suit your needs.
- **I am facing problems with due to headers included using angled
brackets, can I change them to double quotes instead?**
- Yeah that would be okay, anyway the general practice is to add user
header files using double quotes.
- **Using -Werror flag in regexscan causes build to fail, should I
remove it?**
- Since `regexscan` is not the part of default build you can ignore
it.
- **In scheduler source code the preprocessor macro value for
FOSSDB_CONF is different from that in lib, is that correct?**
- We have made some changes, please change it to the same as in lib.
## Conclusion and Further Plans
- Try adding the version and commit hash info.
- Implement writing version files for each build.
- Add proper comments in the `CMakeLists.txt` files.
- Complete the coverage build configuration
- Start implementing the install configurations
## Attendees
- [Michael C. Jaeger](https://github.com/mcjaeger)
- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd)
- [Gaurav Mishra](https://github.com/GMishx)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-4.md
+66
View File
@@ -0,0 +1,66 @@
---
title: Coding Week 3 Meeting
date: 2021-06-22 23:22
tags: [gsoc, FOSSology]
category: gsoc
description: "In this fifth meeting, question related to versioning and obtaining
commit hash were discussed, this was a short meeting."
image: "/images/tech-wallpaper-5.webp"
---
# Coding Week 3 Meeting
In this fifth meeting, question related to versioning and obtaining commit hash were discussed, this was a short meeting.
## Week 3 Progress
> Version file Implementation
>
> - Initial functions on obtaining commit and branch info
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **What is the regex expression used for obtaining version
information?**
- The regex has recently been modified to cover recent versions. The
latest form is as below:
``` cpp
([[:digit:]]+.[[:digit:]]+.[[:digit:]]+)(-?rc[[:digit:]]+)?-?([[:digit:]]*)-?[[:alnum:]]*
```
- You can also try alternatives to regex if possible for CMake.
- **Should I use** `git describe --tags` **or**
`git describe --always HEAD` **for obtaining version information?**
- In FOSSology we always use `git describe --tags`, no exception
whatsoever.
- CMake provides a preset configuration for the install path on GNU
systems, you can see the description
[here](https://cmake.org/cmake/help/v3.10/module/GNUInstallDirs.html)
based on the
[configuration](https://www.gnu.org/prep/standards/html_node/Directory-Variables.html)
suggested by the GNU After comparing the variables defined in
Makefile.conf with these, it seems directly taken from GNU standards.
So I wanted to ask if this would be okay to stick to the presets,
instead of manually declaring the same paths? The former step will
reduce the number of variables we are currently caching and will make
it flexible for different installation scenarios.
- Using the GNU standards is the ideal situation but FOSSology uses
slightly different locations. For example, all agents end up under
`/usr/local/share/fossology/` with their individual folders instead
of going to `/usr/local/bin/`.
- If the same results can be achieved by using the
`CMAKE_INSTALL_<dir>` and `CMAKE_INSTALL_PREFIX` then yeah, it will
be preferred.
## Conclusion and Further Plans
- Try adding the version and commit hash info.
- Implement writing version files for each build.
## Attendees
- [Michael C. Jaeger](https://github.com/mcjaeger)
- [Gaurav Mishra](https://github.com/GMishx)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-5.md
+82
View File
@@ -0,0 +1,82 @@
---
title: Coding Week 4 Meeting-1
date: 2021-06-29 23:22
tags: [gsoc, FOSSology]
category: gsoc
description: "In this seventh meeting question related to installing the FOSSology were discussed."
image: "/images/tech-wallpaper-6.webp"
---
# Coding Week 4 Meeting-1
In this seventh meeting question related to installing the FOSSology were discussed.
## Week 4 Progress
> CMake configuration files have been refactored to make each agent as a separate sub-project.
>
> - Symbolic links are installing.
> - VERSION files can be generated now during configure step
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **There are two types of replacements CMake can configure file with.**
`@VARIABLE@` **and** `${VARIABLE}` **. Since in PHP** `$variable` **is
used, it may create problem for CMake replacements. So may I replace
them?**
- Yeah sure, go ahead. It will be more robust.
- The replacement of `$VARIABLE` can be stopped by using `@ONLY`
option in `configure_file(...)` command.
- **How to generate vendor directory?**
- The code for generating vendor directory is in `src/Makefile`.
- Before executing code for the generation, make sure to copy
`composer.json` and `composer.lock` to the target directory.
- There is also a patch that FOSSology needs to function as intended.
Make sure to run that patch to check and apply.
- For now, we generate *vendor* while building, but it would be nice
if it can be generated in the build step.
- **Currently I am generating the VERSION file in configure step itself.
Should I move it to the build or install step?**
- Yeah, please move it to the build step. As in configure step the
data might be outdated.
- **Is there any configuration for Release that we can use to install or
test?** *(Michael)*
- Yeah, there are 4 inbuilt configurations for various levels of
optimization and can be applied to tests and installation.
- **Is the VERSION file is generated for each agent or whole project at
once? Because in the latter case, the VERSION file can be generated as
the last step.**
- No agent has a VERSION file along with the main VERSION file for
FOSSology.
- **How I can build and install a single agent or component?**
- There are two ways you can build and install a specific agent or
component only.
- The first one is quite simple. Just change your directory to the
specific agent's directory and run all the usual commands for
building and installing.
- The second one is a bit for typing work. This can be used directly
from the top-level directory. After configuring the CMake, you can
run the following command to install the specific component.
``` bash
# for Unix Makefiles
make list_install_component # this will list all the available components
cmake -DCOMPONENT=<component-name> -P cmake_install.cmake
```
- I am writing a macro that will let us install a component by simply
running `make install component`.
## Conclusion and Further Plans
- Implement generation of vendor directory.
- Move VERSION file generation to build step.
## Attendees
- [Michael C. Jaeger](https://github.com/mcjaeger)
- [Gaurav Mishra](https://github.com/GMishx)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-6.md
+58
View File
@@ -0,0 +1,58 @@
---
title: Coding Week 4 Meeting-2
date: 2021-07-02 22:22
tags: [gsoc, FOSSology]
category: gsoc
description: "In this eighth meeting questions related to post install generation were asked. This was a short meeting."
image: "/images/tech-wallpaper-7.webp"
---
# Coding Week 4 Meeting-2
In this eighth meeting questions related to post install generation were asked. This was a short meeting.
## Week 4 Progress
> Version parsing logic implemented.
>
> - VERSION and COMMIT_HASH added to every executables.
> - Installing part is complete except `cli`.
> - Symbolic Links are installing and working fine.
> - Version, Symbolic Links, `VERSION` file generation, `version.php` generation are now more modular and called via a single function for each agent
> - Most dependencies are now moved to single configuration file.
> - Vendor directory generation and installing are now working.
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **Why all the symbolic links in cli points to** `fo_wrapper`
**script?**
- The `fo_wrapper` script calls the PHP script on the symbolic link
that called the fo_wrapper. It also initializes any requirement
before calling the scripts.
- **How to generate all the other configuration in**
`/usr/local/etc/fossology` **directory?**
- You can find the input files for all these configurations in the
`install/defcon` directory.
- **What are** `OBSOLETEFILES` **in** `www/ui/Makefile` **?**
- They are kept for compatibility purposes. Although they have been
removed in the current versions of FOSSology, if a user installs a
new version on top of an older instance, then we should explicitly
remove those files.
- **I have created a separate folder for generating vendor directory. Is
that okay?**
- Yeah, it should be fine, But it would be better to rename it to
something else. Or even better if moved to *www* itself. Since these
files are used by www.
## Conclusion and Further Plans
- Move `vendor` scripts to `www` directory.
- Implement installing for FOSSology cli.
- Implement installing configuration scripts.
- Finish installation for testing
## Attendees
- [Gaurav Mishra](https://github.com/GMishx)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-7.md
+46
View File
@@ -0,0 +1,46 @@
---
title: Coding Week 5 Meeting
date: 2021-07-09 22:22
tags: [gsoc, FOSSology]
category: gsoc
description: "This week was dedicated to perfecting CMake Installation Configuration. The installation was tested and bugs were discussed."
image: "/images/tech-wallpaper-8.webp"
---
# Coding Week 5 Meeting
This week was dedicated to perfecting CMake Installation Configuration. The installation was tested and bugs were discussed.
## Week 5 Progress
> CMake Installation Configuration is almost complete.
>
> - FOSSology can be installed completely via CMake
> - Post install script generation also added
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- There are permission problems while running bash script of `nomos`,
`monk` and `genvendor`.
- One possible fix can be to add `bash` before each bash scripts.
- The other fix is to modify shebang line in each script from
`#!/bin/sh` to `#!/bin/bash`.
- In copyright agent same files are being compiled thrice, this is
slowing down the build.
- I am working on it. The problem is occurring because of three
different executables.
- I will try to combine the common objects together.
- There are some redundant files in the installation. And VERSION file
is missing in `/usr/local/share/fossology`.
## Conclusion and Further Plans
- Fix copyright build.
- Remove redundant files and folders.
- Fix permission issues.
## Attendees
- [Gaurav Mishra](https://github.com/GMishx)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-8.md
+66
View File
@@ -0,0 +1,66 @@
---
title: Coding Week 7 Meeting
date: 2021-07-23 22:22
tags: [gsoc, FOSSology]
category: gsoc
description: "This week I implemented CMake packaging configuration for FOSSology.
There were two meetings in this week and this report covers both of
them."
image: "/images/tech-wallpaper-9.webp"
---
# Coding Week 7 Meeting
This week I implemented CMake packaging configuration for FOSSology. There were two meetings in this week and this report covers both of them.
## Week 7 Progress
> Initial CMake packaging configuration implemented.
>
> - Packages can be built according to the FOSSology previous packaging structure.
> - Copyright, ecc and keyword now builds faster.
> - To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **Where I can find packaging info for FOSSology?**
- All the scripts and companion files are located inside `debian`
folder.
- The most important files are `control`, which contains the
dependency and description of each package, and `rules` file, which
contains the make commands for creating the packages.
- **What are** `${shlibs:Depends}` **and** `${misc:Depends}` ?
- They are dependencies required for creating Debian packages. CMake
should be adding them by default so we can safely ignore them.
- **Will the new packages have the same structure as the old ones?**
*(Michael)*
- Yes for compatibility purposes Gaurav has suggested exactly follow
the same structure as the old one.
- **Copyright build is slow because the same object files are being
compiled three times, can you improve that?** *(Gaurav)*
- I can try compiling the common object files beforehand and then
adding the executables. But how to know the common object files?
- Gaurav showed me where in the Makefiles I can find the common object
files.
- There are problems with copying the symbolic link and packaging them.
So I have to find some alternatives to resolve that.
- With component installing, package description can no longer be set.
- The `fossology-common` package contains file from `fossology-db`
package. And the `fossology-db` package is empty.
- Gaurav said this was unexpected and should not happen. This seems to
be a very old bug with packaging.
## Conclusion and Further Plans
- Work more on the packaging.
- Improve compilation of copyright and monk agents
- Try to solve the packaging bug and add a pull request for that.
- Move on to implementing testing configurations.
## Attendees
- [Michael C. Jaeger](https://github.com/mcjaeger)
- [Gaurav Mishra](https://github.com/GMishx)
- [Anupam Ghosh](https://github.com/ag4ums)
- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd)
- [Avinal Kumar](https://github.com/avinal)
static/content/posts/gsoc/meeting-9.md
+59
View File
@@ -0,0 +1,59 @@
---
title: Coding Week 8 Meeting
date: 2021-07-30 22:47
tags: [gsoc, FOSSology]
category: gsoc
description: "This week I implemented CMake packaging configuration for FOSSology. The
new configuration fixes issue with previous packaging configurations. It
also retains the component wise installation features."
image: "/images/tech-wallpaper-10.webp"
---
# Coding Week 8 Meeting
This week I implemented CMake packaging configuration for FOSSology. The new configuration fixes issue with previous packaging configurations. It also retains the component wise installation features.
## Week 8 Progress
> CMake Packaging configuration almost completed.
- Packages can be built according to the FOSSology previous packaging structure.
- Initial testing configuration added.
- Ninja build has been fixed.
- To test the current progress, follow the instructions [here](https://github.com/avinal/FOSSology/wiki#test-the-new-system-only-gcc-with-make-and-ninja-tested-for-now)
## Discussions
- **How is the testing implemented in FOSSology?**
- Not all agents have testing implemented.
- There are two types of tests *Unit* and *Functional*.
- At first, the test executable calls multiple PHP scripts to create a
test environment. And then tests are executed.
- Files related to testing and common for all the agents are in
`src/testing`
- Other tests depends on `phpunit`. This *PHPUnit* is generated inside
`vendor`.
- **As of now, the testing configurations are hardcoded, what should I
do, because it seems the testing configuration will require changes to
a lot of files?**
- Decide a deadline for the testing configuration and if until that
point there is not very productive implementation then move to the
next task that is implementing CI.
- As of now building, installation, and packaging via CMake is working
and in a stable state. To create an initial Pull Request. This would
also be useful in case of the final evaluation and further testing
will be based on this PR itself.
- Fix any bugs or if there is the scope of improvement in Building,
Installation and Packaging do that.
## Conclusion and Further Plans
- Prepare for an initial PR.
- Fix known bugs and apply Improvements.
- Work on testing configurations.
## Attendees
- [Gaurav Mishra](https://github.com/GMishx)
- [Shaheem Azmal M MD](https://github.com/shaheemazmalmmd)
- [Avinal Kumar](https://github.com/avinal)