avinal/avinal.github.io
1
0
Fork 0
mirror of https://github.com/avinal/avinal.github.io.git synced 2026-07-04 07:40:09 +05:30
Code Packages Releases Activity

move articles to static

Browse Source 
Signed-off-by: Avinal Kumar <avinal.xlvii@gmail.com>
This commit is contained in:
Morumotto
2023-01-24 16:40:54 +05:30
parent 43e157885f
commit 9a88c34745
12 changed files with 40 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
static/content/posts/development/final-evaluation.md
+831
View File
@@ -0,0 +1,831 @@
---
title: Google Summer of Code 2021
date: 2021-08-19 23:07
tags: [gsoc, FOSSology]
category: development
description: This is the final report of my Google Summer of Code 2021 at The FOSSology Project.
image: "/images/gsoc-wall.webp"
---
# Google Summer of Code 2021
<style>
.rd {color:red;font-weight:bold}
.gr{color:green;font-weight:bold}
.or{color:orange;font-weight:medium}
ul{margin-bottom:0}
</style>
## The CMake Build system
FOSSology is quite an old and mature project. The project has been using
bare metal **Makefiles**. As the project is growing with new agents and
modernization it was required to have a modern build system.
The FOSSology is a suite of well-integrated and synchronized
sub-projects (called agents) written in C, C++, and PHP. Most of the
major agents are in C, C++ and that made CMake an obvious choice for a
new build system for FOSSology. CMake is a versatile set of build, test,
and packaging tools and is the most popular choice of C/C++ developers.
CMake can be extended to create a build system for other languages too
via custom scripts.
## GitHub Actions CI/CD
<img src="/images/ci.webp"
class="float-md-right rounded border border-info ml-3 float-md-right rounded border border-info ml-3"
width="350" alt="A CI Meme" />
Since the FOSSology project moved on Github, it has used only the free
Travis CI service for OSS projects. At the time of writing Travis CI has
reduced its free tier CI services. GitHub Actions provides faster
builds. Since GitHub Actions is a fully managed service by GitHub, we
don't need to know how to scale and operate the infrastructure to run
it.
It is straightforward to use with GitHub because when we fork a
repository, the actions automatically get forked. This allows you to
test and build projects very efficiently and even run them closer to the
developer. Also, you have readily available access to the GitHub API,
making it more popular among developers.
## Improvements over previous build system and CI
The new build system and CI brings a lot of improvements and features.
The list below describes them.
- CMake enforces out-of-source builds. This was already possible with
the previous build system but not a strict requirement. This feature
keeps the source code clean and makes cleaning the build artifacts
easy. (Just remove the build folder :)
- One of the major improvements over the previous build system is faster
build times. CMake generates parallel build-enabled configurations for
all generators. In our tests, the new build system is at least twice
as fast as the previous one. With further improvement in
configuration, we will be able to further cut the build times.
- Previously FOSSology can only be built using *Unix Makefiles*. With
CMake, we can now use many other popular generators such as *Ninja*.
- Now it is also very flexible to choose different compilers. This will
help support more platforms and architecture in the future. As of now,
we are experimenting with Clang compilers.
- FOSSology is quite an old project and a lot of agent testing code was
written in the last decade. Initially, none of them were compatible
with the new build system, but we were able to hack most of the test
code using better-improved methods. Test times have also improved.
- Migrating from Travis CI to GitHub Actions was another big move and
for the most part, it removes the dependency on a third-party CI
service. Along with that GitHub Actions provides better build times,
tons of new features, and better integration with other GitHub
services.
## Deliverables
- Final Pull Request [#2075](https://github.com/fossology/fossology/pull/2075)
- Pull Request Branch [avinal/feat/buildsystem](https://github.com/avinal/fossology/tree/avinal/feat/buildsystem)
- Working Branch (individual commits)
- [avinal/feat/cmake-buildsystem](https://github.com/avinal/fossology/tree/avinal/feat/cmake-buildsystem)
- [avinal/feat/testing](https://github.com/avinal/fossology/tree/avinal/feat/testing)
- Project Issue [#1913](https://github.com/fossology/fossology/issues/1913)
- Project Discussion [#1931](https://github.com/fossology/fossology/discussions/1931)
- Weekly Reports
- [Personal Blog](https://gsoc.avinal.space)
- [FOSSology Official Blog](https://fossology.github.io/gsoc/docs/2021/buildsystem/)
### CMake Build System Tasks
<table>
<colgroup>
<col style="width: 7%" />
<col style="width: 23%" />
<col style="width: 15%" />
<col style="width: 15%" />
<col style="width: 30%" />
<col style="width: 23%" />
<col style="width: 38%" />
</colgroup>
<thead>
<tr class="header">
<th>#</th>
<th>Agents</th>
<th>Build</th>
<th>Install</th>
<th>Testing</th>
<th>Packaging</th>
<th>Remarks</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>1</td>
<td>adj2nest</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>2</td>
<td>buckets</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><blockquote>
<p><span class="gr">YES</span></p>
</blockquote></td>
<td></td>
</tr>
<tr class="odd">
<td>3</td>
<td>cli</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>4</td>
<td>copyright</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>5</td>
<td>debug</td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>6</td>
<td>decider</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>7</td>
<td>deciderjob</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>8</td>
<td>delagent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">Functional</span></li>
<li><span class="rd">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>9</td>
<td>demomod</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td><ul>
<li><span class="or">Functional</span></li>
<li><span class="or">Unit</span></li>
</ul></td>
<td><span class="or">NO</span></td>
<td><em>(Not Used)</em></td>
</tr>
<tr class="even">
<td>10</td>
<td>example_wc_agent</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td><ul>
<li><span class="or">Functional</span></li>
<li><span class="or">Unit</span></li>
</ul></td>
<td><blockquote>
<p><span class="or">NO</span></p>
</blockquote></td>
<td><em>(Not Used)</em></td>
</tr>
<tr class="odd">
<td>11</td>
<td>clib</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>12</td>
<td>cpplib</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>13</td>
<td>phplib</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>1 functional test needs fix</td>
</tr>
<tr class="even">
<td>14</td>
<td>maintagent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>15</td>
<td>mimetype</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>16</td>
<td>monk</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>17</td>
<td>ninka</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td><ul>
<li><span class="or">Functional</span></li>
<li><span class="or">Unit</span></li>
</ul></td>
<td><span class="or">NO</span></td>
<td><em>(Deprecated)</em></td>
</tr>
<tr class="even">
<td>18</td>
<td>nomos</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>19</td>
<td>ojo</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>1 functional test needs fix</td>
</tr>
<tr class="even">
<td>20</td>
<td>pkgagent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>21</td>
<td>readmeoss</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>22</td>
<td>regexscan</td>
<td><span class="or">YES</span></td>
<td><span class="or">YES</span></td>
<td></td>
<td><blockquote>
<p><span class="or">NO</span></p>
</blockquote></td>
<td><em>(Deprecated)</em></td>
</tr>
<tr class="odd">
<td>23</td>
<td>reportImport</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>24</td>
<td>reuser</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>25</td>
<td>reso</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>26</td>
<td>scheduler</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">Functional</span></li>
<li><span class="rd">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>Tests needs fix</td>
</tr>
<tr class="odd">
<td>27</td>
<td>softwareHeritage</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="even">
<td>28</td>
<td>spasht</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>29</td>
<td>spdx2</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>1 Test failing in CI</td>
</tr>
<tr class="even">
<td>30</td>
<td>unifiedreport</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>31</td>
<td>ununpack</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="rd">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td>Unit tests needs fix</td>
</tr>
<tr class="even">
<td>32</td>
<td>wget_agent</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="gr">Functional</span></li>
<li><span class="gr">Unit</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
<tr class="odd">
<td>32</td>
<td>www</td>
<td><span class="gr">YES</span></td>
<td><span class="gr">YES</span></td>
<td><ul>
<li><span class="rd">UI</span></li>
</ul></td>
<td><span class="gr">YES</span></td>
<td></td>
</tr>
</tbody>
</table>
### GitHub Actions CI Tasks
| # | CI Tasks | Status |
|-----|-----------------------------------------|----------------------------------------------------------|
| 1 | <span class="gr">build</span> | Added Ubuntu 20.04 GCC 8, 9 and Clang, GCC 7 not working |
| 2 | <span class="gr">c/cpp unit test</span> | Added, delagent, scheduler and ununpack not working |
| 3 | <span class="gr">phpunit tests</span> | Added, delagent and scheduler functional not working |
| 4 | <span class="rd">cahching</span> | Not implemented |
| 5 | <span class="rd">source install</span> | Not implemented |
(<span class="gr">GREEN</span>: COMPLETED, <span class="rd">RED</span>:
INCOMPLETE, <span class="or">ORANGE</span>: NOT NEEDED/DEPRECATED)
## How does it work and how to use it?
The new build system retains the modular and hierarchical structure of
the previous build system. On the other hand, the new build system
provides several new flags to control the build. The new build system
forces out-of-source build instead of the previous in-source builds.
This keeps the source clutter-free and reduces the chance of
accidentally deleting source files. *Testing still needs some in-source
artifacts, this will be solved once all the tests are fixed according to
the new build system.*
Each agent is a complete CMake sub-project with its independent
configuration for building, installing, and testing. That means a single
agent can be built and installed separately and even removed from the
default build without breaking other builds. The directory structure is
as below.
```bash
.
├── build # temporary directory for build artifacts
├── cmake # CMake modules for FOSSology
│ ├── FoPackaging.cmake # CMake Packaging configurations
│ ├── FoUtilities.cmake # Custom CMake utilities
│ ├── FoVersionFile.cmake # VERSION version.php CMake template file
│ ├── SetDefaults.cmake # CMake defaults for this project
│ ├── TestInstall.make.in # Template makefile for install during tests
│ └── VERSION.in # VERSION file template
├── src
│ ├── agent-1 # Agent sub-project
│ │ ├── agent # Agent's source code directory
│ │ │ ├── agent-source-code
│ │ │ └── CMakeLists.txt
│ │ ├── agent_tests # Agent's test directory
│ │ │ ├── Unit
│ │ │ ├── Functional
│ │ │ └── CMakeLists.txt
│ │ ├── ui # Agent's UI source code
│ │ │ ├── templates
│ │ │ └── agent-ui-code
│ │ └── CMakeLists.txt # Agent's top-level CMake configuration
: :
│ ├── other agents
: :
│ └── CMakeLists.txt # Source intermediate CMake configuration
:
├── other directories and files
:
└── CMakeLists.txt # FOSSology Top-level CMake configuration
```
The `cmake` directory contains customized CMake modules and templates
for FOSSology. This directory is required for all the operations. The
general workflow of the new build system as well as how to use it is
described below.
1. Since the new build system is still in review. You must fork
FOSSology and pull the
[#2075](https://github.com/fossology/fossology/pull/2075) pull
request branch. Once you are in FOSSology root, run these commands.
```bash
git fetch https://github.com/avinal/fossology avinal/feat/buildsystem:buildsystem
git checkout buildsystem
```
2. The first step towards building is to create a temporary directory
for storing intermediate files and build artifacts. By convention we
use a directory named `build`, but you can use any name. (**NOTE:
For testing do not use other names**)
```bash
mkdir build
cd build
```
3. In the next steps, we will configure the CMake project and generate
the required configurations. You can use several flags to control
the build. Given below are the flags available for this project.
<table style="width:99%;">
<colgroup>
<col style="width: 34%" />
<col style="width: 43%" />
<col style="width: 20%" />
</colgroup>
<thead>
<tr class="header">
<th>CMake Flags</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>-DCMAKE_INSTALL_PREFIX=&lt;path&gt;</strong></td>
<td>Sets the install prefix.</td>
<td><code>/usr/local</code></td>
</tr>
<tr class="even">
<td><strong>-DAGENTS="agent1;agent2..."</strong></td>
<td>Only configure these agents.</td>
<td>ALL AGENTS</td>
</tr>
<tr class="odd">
<td><strong>-DOFFLINE=&lt;ON/OFF&gt;</strong></td>
<td>Controls vendor generation, ON=NO</td>
<td><strong>OFF</strong></td>
</tr>
<tr class="even">
<td><p><strong>-DCMAKE_BUILD_TYPE=&lt;type&gt;</strong></p>
<blockquote>
<ul>
<li>Controls build type aka level optimisation</li>
</ul>
</blockquote></td>
<td><ul>
<li><code>Debug</code></li>
<li><code>Release</code></li>
<li><code>RelWithDebInfo</code></li>
<li><code>MinSizeRel</code></li>
</ul></td>
<td><code>Debug</code></td>
</tr>
<tr class="odd">
<td><strong>-DTESTING=&lt;ON/OFF&gt;</strong></td>
<td>Controls testing config generation</td>
<td><blockquote>
<p><strong>OFF</strong></p>
</blockquote></td>
</tr>
<tr class="even">
<td><strong>-DMONOPACK=&lt;ON/OFF&gt;</strong></td>
<td>Package adj2nest and ununpack seperately</td>
<td><strong>OFF</strong></td>
</tr>
<tr class="odd">
<td><strong>-GNinja</strong></td>
<td>Use Ninja instead of Unix Makefiles</td>
<td><em>Unix MakeFiles</em></td>
</tr>
</tbody>
</table>
There are lots of inbuilt CMake command-line options you can see
them in the official
[documentation](https://cmake.org/cmake/help/v3.10/manual/cmake.1.html).
Once you have chosen your flags we can now configure the project
using the following commands.
```bash
# From build folder
cd <name-of-build-directory>
cmake <flags> ..
```
4. The next step is to build the project. You can use parallel jobs to
build faster. For more options you can type `cmake --help` or
`make --help` or `ninja --help`.
```bash
# Common build command for all generators,
# Default number of parallel builds depends on generator used
cmake --build . --parallel <no-of-processes>
# For Unix Makefiles, no parallel build by default
make -j <no-of-processes>
# For Ninja, 8+ parallel build by default (depends on system)
ninja -j <no-of-processes>
```
5. Installing is also as easy as building. You can choose to install
only certain components even if you have built the whole project. If
you directly invoke the install command without building the
project, it will automatically build the project first.
```bash
# For Unix Makefiles
make install
# For Ninja
ninja install
```
6. While testing has some issues, most of the testing is working fine.
For now, you must build and run any test from the FOSSology root
directory only. You can choose to configure a single agent if you
want to test one agent only. See `ctest --help` for controlling test
runs.
```bash
# Common testing command
ctest --parallel <no-of-processes>
# For Unix Makefiles
make test
# For Ninja
ninja test
```
7. You can package FOSSology, the packaging currently lacks copyright
and conf files. But for testing purposes, you can use the following
commands. Similar to installing, if you run the package command
without building the project, it will automatically build the
project first. See `cpack --help` for more packaging options.
```bash
# Common testing command
cpack
# For Unix Makefiles
make package
# For Ninja
ninja package
```
## Known Issues and Drawbacks
Although the transition from Makefiles to CMake and Travis CI to GitHub
Actions is almost complete and working as expected. But it is not free
of drawbacks and issues. This section outlines the known issues at the
time of writing.
![A Bug Meme](https://imgs.xkcd.com/comics/conference_question.png)
- Coverage builds may fail with linking errors.
- Packaging prefix is the same as the install prefix. This requires the
developer to set the install prefix manually before packaging to
produce packages with the correct directory structure.
- Testing and packaging must be used from the FOSSology root directory.
Not doing so may or may not configure the project as intended.
- Previously tests were written hardcoded for the Makefiles. But new
build system requires all artifacts to be generated in a separate
directory. This required me to add symbolic links wherever a generated
script or file is expected. Tests can still leave some artifacts
inside source folders.
- There is no easy way to install a particular agent from the FOSSology
root directory.
- Packages don't contain copyright, readme, and license files. CMake
doesn't provide a way to include these files. This is being tracked by
issue
[#21832](https://gitlab.kitware.com/cmake/cmake/-/issues/21832).
- While packaging the symbolic links may or may not be dereferenced and
hence results in copying the folder too in the target directory.
- Running tests locally may require switching to `fossy` user.
- While configured for testing, it may give permission errors.
- Scheduler, Ununpack, and Delagent unit and functional tests are not
working. I have added an issue
[#2084](https://github.com/fossology/fossology/issues/2084) to track
the progress on fixing these tests.
- CMake doesn't generate uninstall targets. The closest thing to
uninstall is [this snippet](https://gitlab.kitware.com/cmake/community/-/wikis/FAQ#can-i-do-make-uninstall-with-cmake).
This will be later added to the FOSSology.
## Challenges Faced
While this whole project was challenging, some aspects of it were
unforeseen and more challenging. When I decided to go on with this
project I just had enough CMake knowledge to write a configuration for a
very small project. I had never used CMake on this big scale. On the
other side, the FOSSology community is largely unknown to CMake so for
all of us it was learned, practiced, and implement. With support from
mentors, I was able to overcome this challenge with flying colors.
The other challenge was to understand the old build system, how they are
all connected and what is the flow. The complexity can be imagined by
the fact that the most of code and configurations were written in the
decade before the last decade and haven't changed much since then.
The most challenging task was to make tests work with the new build
system. Since tests were mostly hardcoded and the new build system
refactored many of the files and directory, the tests were failing
initially. The testing part took me the most time. All thanks to my
mentor Gaurav, I was able to hack them to suit the
new build system.
## Related Resources and Links
- Fix FOSSology agent tests issue
[#2084](https://github.com/fossology/fossology/issues/2084)
- feat(CI): Migrate API docs generation and deployment to GitHub Actions
pull request
[#1917](https://github.com/fossology/fossology/pull/1917)
- feat(CI): Migrate Static Checks and Analysis to GitHub Actions from
Travis CI [#1919](https://github.com/fossology/fossology/pull/1919)
## Future Development Plans
There is a lot to do with the new build system and CI and it will
probably take a year or to reach a maturity point. I was able to meet
most of the goals but some of them are remaining.
- Fix the tests, probably renovate them from the ground up.
- Find a hack for packaging problems.
- Improve and optimize the build.
- Modernise the source code, remove old, bloated code and replace them
according to new standards.
## What did I learn from this project?
This Google Summer of Code was the busiest time of my life for all good
reasons. I learned a lot about license compliance and how it all works
in the software industry. The next big thing is CMake. As I mentioned I
was just a novice user of CMake. Now I am confident that given any other
large project I will be able to migrate it/improve it. I got to learn
PHP, of which I did not know a single word before GSoC. And finally, I
learned about packing and testing. I had these courses but implementing
them myself and fixing them was a wholesome experience.
Other than that I improved on my communication and presentation skills.
Collaborating with fellow participants was one of the great things that
happened during GSoC.
## Acknowledgments
Google Summer of Code is the best thing that has happened to me this
year so far. Although there are numerous people to say thanks to, I want
to mention key people who were my motivation and support during this
period.
First of all, I want to thank and appreciate my mentors [Gaurav
Mishra](https://github.com/GMishx), [Michael C.
Jaeger](https://github.com/mcjaeger), [Anupam
Ghosh](https://github.com/ag4ums), and [Shaheem Azmal M
MD](https://github.com/shaheemazmalmmd). Without the help and support
from them, all this would not have been possible. They are very polite,
knowledgeable, and helpful.
Finally, I want to thank, my family and friends. I got to meet many
awesome developers as my fellow participants from around the world, I
wish we will do more collaboration in the future.
static/content/posts/development/i-am-loving-it-redhat.md
+119
View File
@@ -0,0 +1,119 @@
---
title: I am loving it! RedHat
date: 2022-02-25 20:47
category: development
tags: [kubernetes, redhat, docker, golang, tekton, openshift, intern]
description: I made it to the Red Hat as a DevTools intern. This post is about onboarding and how I prepared myself for working on the actual project.
image: "/images/fedora-wall.webp"
---
# My internship at Red Hat
I have been contributing to open source for the last 3 years and Red Hat
was one of the companies that I was very fond of. I must say all my
contributions and consistency paid off, and I made it to the Red Hat as
a DevTools intern. This post is about onboarding and how I prepared
myself for working on the actual project.
On the first day of my internship, I met two amazing teammates
[Saytam](https://github.com/) and [Utkarsh](https://github.com/). We
were also introduced to a Senior Software Engineer [Piyush
Garg](https://github.com) who later mentored us. The initial few
meetings were more on the introduction and what to learn topics. Before
I jump into more details let me explain first what does a **DevTools
Developer/Engineer** do?
## What does a DevTools Developer/Engineer do?
From [MDN Web
Docs](https://developer.mozilla.org/en-US/docs/Glossary/Developer_Tools)
**Developer tools (or "development tools" or short "DevTools") are
programs that allow a developer to create, test, and debug software.**
At RedHat, a lot of open source developer tools of industry standards
are developed. There are many, OpenShift, Tekton, CodeReady containers,
and many more.
## Learning on the Go
There was a lot of learning and still a lot to learn. In a meeting with
my manager Pradeepto Bhattacharya, I was told that I will be working on
TektonCD or OpenShift Pipelines, and both of them require a sound
knowledge of Golang, CI/CD, Containers, Docker, and Kubernetes. I was
familiar with CI/CD, containers, and Docker but never used Golang and
Kubernetes. We were provided plenty of good resources and my teammates
also helped with many awesome resources. I am listing all the resources
with their category.
### Go Programming Language
One of Golang's biggest advantages is that it offers the clarity and
ease of use that other languages lack. Golang's advantages make it easy
for new programmers to quickly understand the language and for seasoned
veterans to easily read each other's code.
- [Official Go Documentation](https://go.dev/doc/) - *Start from here*
- [Go by Example](https://gobyexample.com/) - *bite-size examples for
most of the golang features*
- [Golang tutorial series -
GOLANGBOT.COM](https://golangbot.com/learn-golang-series/) - *in-depth
tutorial of golang*
- [Effective Go](https://go.dev/doc/effective_go) - *writing good golang
programs*
- [The Go Playground](https://go.dev/play/) - *official online golang
ide*
- [The Go Programming Language - Book](https://www.gopl.io/) *for
learning advanced level golang*
- [Golang Tutorial for Beginners | Full Go Course - TechWorld with
Nana](https://youtu.be/yyUHQIec83I) *if you prefer video tutorials, I
don't :)*
### Docker
Docker takes away repetitive, mundane configuration tasks and is used
throughout the development lifecycle for fast, easy, and portable
application development - desktop and cloud. Docker's comprehensive
end-to-end platform includes UIs, CLIs, APIs, and security that are
engineered to work together across the entire application delivery
lifecycle.
![The Docker Architecture](/images/docker-architecture.webp)
- [Docker and Containers -
Katacoda](https://www.katacoda.com/courses/docker) *interactive
lessons on docker and containers*
- [Docker for beginners](https://docker-curriculum.com/)
- [Docker Tutorial for Beginners | TechWorld with
Nana](https://youtu.be/3c-iBn73dDE) *video tutorial*
### Kubernetes
![Kubernetes tech](/images/kubernetes-meme.webp)
**Kubernetes** is the Greek word for a ship's captain. We get the words
Cybernetic and Gubernatorial from it. The Kubernetes project focuses on
building a robust platform for running thousands of containers in
production.
- [Learn Kubernetes -
Katacoda](https://www.katacoda.com/courses/kubernetes) *interactive
lessons with kubernetes*
- [kube by example](https://kubebyexample.com/) *learn by doing*
- [Kubernetes Tutorial for Beginners](https://youtu.be/X48VuDVv0do)
*video tutorial*
## *Not so Minimal* Tekton Server
In late January, we were asked to implement our learnings and deep dive
into Kubernetes and TektonCD through an assignment project. Soon we
realized that whatever we were learning so far was not even close to
what we were going to implement. We were given a document containing the
requirements of the applications we were supposed to create along with
all the documentation and architectural diagrams.
The application was called **Minimal Tekton Server**. It is a set of
three different applications, i.e a server, a CLI, and a dashboard. In
short, this application is supposed to *listen to custom resources being
created and then transfer the request to Tekton API to create the
corresponding resource on the OpenShift/Kubernetes cluster.*
So are you interested in how it went? Please follow up with my [next
blog](https://avinal.space/posts/development/lovely-dangerous-things-redhat.html).
static/content/posts/development/lovely-dangerous-things-redhat.md
+213
View File
@@ -0,0 +1,213 @@
---
title: Developing Minimal Tekton Server
date: 2022-02-27 20:47
modified: 2022-03-07 22:47
category: development
image: "/images/development.webp"
tags: [tekton, go, kubernetes, openshift, redhat, intern, golang, openshift-pipelines]
description: 'This blog is a descreptive account of the development of Minimal Tekton Server. This is highly technical in nature, so please make sure that you have sufficient knowledge about Golang, Docker, Kubernetes and TektonCD. You can refer to my [previous blog]("https://avinal.space/posts/development/i-am-loving-it-redhat.html") to know about these topics.'
---
# Developing Minimal Tekton Server
This blog is a descreptive account of the development of Minimal Tekton Server.
This is highly technical in nature, so please make sure that you have sufficient
knowledge about Golang, Docker, Kubernetes and TektonCD. You can refer to my
[previous blog]("https://avinal.space/posts/development/i-am-loving-it-redhat.html")
to know about these topics.
As mentioned in my last blog, we were given to implement an application
named **Minimal Tekton Server**. The problem statement reads:
> We will be designing and implementing an application that will be
> talking to Tekton APIs to create resources on a Kubernetes/OpenShift
> Cluster. The application will expose some fields of the Tekton
> Resources which the user will provide and then this application will
> create Tekton resources by talking to Tekton APIs available on the
> cluster to create the resources based on the user-provided fields.
There are three parts in this project for the application and two more
parts for the CI/CD using TektonCD and Kubernetes/OpenShift. I will go
through each part descriptively and try to explain what we did.
## The Architecture of MKS
The first task in the development of the Minimal Tekton Server was
creating its architectural diagram. Our first diagram was trash compared
to the final diagram. Yeah, we learned. I will be explaining our
final(obviously) architectural diagram and try to make some sense out of
band-aids and duct tapes.
<img src="/images/mks-architecture.webp"
class="img-fluid my-3 img-fluid my-3" alt="The MKS Arhitecture" />
Let me start with explaining **What are MKS Resources?**. I hope you
know at least tidbits about Kubernetes and by the definition: *A
resource is an endpoint in the Kubernetes API that stores a collection
of API objects of a certain kind; for example, the built-in
:code:\`pods\` resource contains a collection of Pod objects.* But
developers soon realized that these in-built resources were not enough
for the ever-growing applications of Kubernetes. Here [custom
resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
comes into the picture. *A custom resource is an extension of the
Kubernetes API that is not necessarily available in a default Kubernetes
installation.* To define a custom resource we use something called
[Custom Resource
Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/).
So MKS Resources are such custom resources that correspond to the
TektonCD custom resources.
<img src="/images/venus-flytrap.gif"
class="float-md-right ml-3 float-md-right ml-3" width="250"
alt="A venus flytrap engulphing an insect." />
Let us now focus on the box containing `Controller` and `API server`.
The controller can be said as a stimulus-response mechanism. Take the
analogy of a Venus Flytrap plant. The trap is initially open. There are
`trigger` hairs on the inside of the trap. Once an insect is detected,
there is a change of state and the trap closes in a blick on the eye.
The controller works the same way. It listens for the change in the
state of the MKS resources and immediately transfers the request to the
Tekton API to reflect the change in the corresponding Tekton resources.
The changes can be creation, deletion, or updating. The API server
ensures that there is a working connection between our controller and
the Tekton API.
MKS Server also exposes APIs to introduce a change of state in the MKS
resources. In technical terms these are called `verbs`. There are five
such verbs that we have exposed: `create`, `update`, `get`, `delete`,
and `list`. They can be utilized by a REST client, or in our case **MKS
CLI** to introduce desired change. The MKS command-line interface
provides commands and subcommands to do the desired tasks.
Whenever there is a change in the state, there is a logic running inside
the controller to react on that and that also affects our database. We
store four datapoints in our database: `created`, `deleted`,
`completed`, and `failed`. They tell us about the current statistcs of
our MKS resource using a single-page web app called **MKS Dashboard**
(or UI).
This was about the architecture of the Minimal Tekton Server. Let us
jump into more technical stuff.
## How to implement a CRD controller?
During this assignment, something that took the most time and effort was
the implementation of a controller for our custom resources. This isn't
very hard if you go by the rules and do the things according to the
well-defined documents and blogs since this is a standard step in the
implementation of any custom resource controller. But did we follow the
rules? Hell no! But this time, let us go step-by-step.
1. The first step is to define a `CustomResourceDefinition` for our custom
resource. Let us define a CRD called `spacetime`. To do this you can write a
YAML file like below.
```yaml
# file: spacetime-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: spacetimes.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1alpha1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
message:
type: string
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: spacetimes
# singular name to be used as an alias on the CLI and for display
singular: spacetime
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: SpaceTime
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- st
```
You can learn more about the fields and options
[here](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/).
The CRD that we defined above corresponds to the `CustomResource` given
below. Once you apply the above file you will be able to see the
`spacetime` custom resource on your Kubernetes/OpenShift cluster.
```yaml
# file: spacetime-cr.yaml
apiVersion: spacetimes.example.com/v1alpha1
kind: SpaceTime
metadata:
name: spacetime-cr
spec:
message: "Hello from space!"
```
Apply them using the following commands:
```bash
kubectl apply -f spacetime-crd.yaml
kubectl apply -f spacetime-cr.yaml
```
1. Once we have defined our custom resources, we need to define the
types that will correspond to this custom resource definition. This
can be done using `k8s.io/apimachinery/pkg/apis/meta/v1` package
written in golang. Did I tell you that this is all in golang? Well,
now you know. Create a package structure for a golang project and
add the definition of the type as given below.
```bash
mkdir -p pkg/api/spacetime/v1alpha1
touch pkg/api/spacetime/v1alpha1/{spacetime_types,register,doc}.go pkg/api/spacetime/register.go
```
Add the following content to the corresponding files.
```go
// file: /pkg/api/spacetime/v1alpha1/spacetime_types.go
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type SpaceTime struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec SpaceTimeSpec `json:"spec"`
}
type SpaceTimeSpec struct {
Message string `json:"message"`
}
type SpaceTimeList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata"`
Items []SpaceTime `json:"items"`
}
```
### Attribution
- Photo by [Luca Bravo](https://unsplash.com/@lucabravo?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/photos/XJXWbfSo2f0?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText).
static/content/posts/development/wakatime.md
+211
View File
@@ -0,0 +1,211 @@
---
title: How I implemented WakaTime embeddable Coding Graph GHA?
date: 2021-02-02 21:47
tags: [wakatime, github-action, coding]
category: development
image: "/images/waka.webp"
description: 'If you use WakaTime to track your coding activity. You can add that to
your README as a bar graph or embed it in your blog/portfolio. Just add this
action to any of your repositories and there you have it.'
---
# How I implemented WakaTime embeddable Coding Graph GHA?
If you use WakaTime to track your coding activity. You can add that to
your README as a bar graph or embed it in your blog/portfolio. Just add this
action to any of your repositories and there you have it.
## Implementation Details
This GitHub Action is divided into three parts. I didn't want to use
Docker but it seems it doesn't work well without it. Let dive a little
into technical details. Three parts are as below.
1. [main.py](https://github.com/avinal/Profile-Readme-WakaTime/blob/master/main.py)
python script. This script contains many procedures.
- [Getting JSON data file via WakaTime
API](https://github.com/avinal/Profile-Readme-WakaTime/blob/master/main.py#L52)
```python
def get_stats() -> list:
...
return data_list
```
This function parses the JSON file received and scraps out the useful
data as a list of lists. Data scraped are language list, time spent on
each language, percentage of the time, start date, and end date. For
this action, I have limited the number of languages to 5 however it
should be very easy to increase that number.
- [Setting the
Timeline](https://github.com/avinal/Profile-Readme-WakaTime/blob/master/main.py#L13)
```python
def this_week(dates: list) -> str:
...
return f"Coding Activity During: {week_start.strftime('%d %B, %Y')} to {week_end.strftime('%d %B, %Y')}"
```
The start date and end date scraped in the last function are used here
to set the timeline. Because date in JSON is provided in UTC as below
:
```json
date: "YYYY-MM-DDTHH:MM:SSZ"
```
I striped it to simple dates only. We can set them manually by taking
the current time from the system. But that method is flawed. But this
method ensures that JSON was received latest and the request was
successful. Any anomaly will point to a failure in request.
- [Creating a bar
graph](https://github.com/avinal/Profile-Readme-WakaTime/blob/master/main.py#L21)
```python
def make_graph(data: list):
...
savefig(...)
```
Lastly, it is time to generate the graph and save them as an image.
This function uses the data scraped in the first step. Creating a bar
graph using matplotlib is easy.
Decorating was a bit difficult. I wanted this graph to merge with
GitHub's look so I chose to color the bar as GitHub colors the
languages. That data is stored as colors.json. Many of the languages have
slightly different spelling in GitHub as compared to WakaTime. So some
languages are shown in default color. That can be improved if we
notice that language and change its color manually. Lastly, the graph
is saved both as SVG and PNG. SVGs are better to put on a responsive
page.
2. [entrypoint.py](https://github.com/avinal/Profile-Readme-WakaTime/blob/master/entrypoint.sh)
shell script. This shell script clones the repository copies the
image and pushes changes to the master. There were several problems.
First of all authentication. This was solved by using a remote
repository address using GitHub Token. And it seems that GitHub
doesn't allow to commit without a username and email. So I used
**github-actions** bot email.
```bash
remote_repo-"https://${GITHUB_ACTOR}:${INPUT_GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config user.name "GitHub Actions"
```
`41898282` is the id assigned to the github-actions bot. Don't ask
where I found them 🙂.
Another problem was to separate repository name from combined
*username/repository-name* provided by `${GITHUB_REPOSITORY}`. GitHub doesn't
provides a direct way to get just the repo name. We used *Internal
Field Separator*. It returns an array and works similar to `split()`
command in Python and Java.
```bash
# '/' is the seperator
IFS-'/' read -ra reponame <<< "${GITHUB_REPOSITORY}"
# returned {username, repository}
repository-"${reponame[1]}"
```
After that, all other commands are pretty straight. Commit the added
files and push them.
3. [Dockerfile](https://github.com/avinal/Profile-Readme-WakaTime/blob/master/Dockerfile)
**IMPORTANT** It took a lot of time to reach this state 🥱. This is
where all the magic happens. I am running <span
class="title-ref">ubuntu:latest</span> inside the container. I first
update the distribution. Then install the required python packages.
Lastly, I invoke the python script and shell script.
There was an almost impossible problem, I searched hundreds of posts
that *how can I access the generated files inside Docker container*, but
no luck. But at last, I found a workaround(obviously otherwise you
wouldn't be reading this by now 🤣) each command is run in a separate
virtual sub-container. As the command ends its output is also lost but
not when you club multiple commands together. At least not until every
command is finished. The generated files are available to the next
clubbed process. I did that by combining the python script run and shell
script run.
```dockerfile
CMD python3 /main.py && /entrypoint.sh
```
This part is the smallest yet took the most time and tries while
developing this action.
## How to use this GitHub Actions?
1. First get your WakaTime API Key. You can get it from your
[WakaTime](<https://wakatime.com>) account settings.
2. Save WakaTime API Key to Repository Secret. Find that by clicking
the Settings tab. Keep the name of the secret as
**WAKATIME_API_KEY**.
3. Add the following line in your README.md of your repo.
```html
<img src="https://github.com/<username>/<repository-name>/blob/<branch-name>/images/stat.svg" alt="Alternative Text"/>
Example: <img src="https://github.com/avinal/avinal/blob/main/images/stat.svg" alt="Avinal WakaTime Activity"/>
```
> You can use this method to embed in web pages too. **Do not use the
> markdown method of inserting images. It does not work sometimes.**
4. Click the **Action** tab and **choose to set up a workflow
yourself**.
5. Copy the following code into the opened file, you can search for
**WakaTime Stat** in the marketplace tab for assistance.
```yaml
name: WakaTime status update
on:
schedule:
# Runs at 12 am '0 0 * * *' UTC
- cron: "1 0 * * *"
jobs:
update-readme:
name: Update the WakaTime Stat
runs-on: ubuntu-latest
steps:
# Use avinal/Profile-Readme-WakaTime@<latest-release-tag> for latest stable release
# Do not change the line below except the word master with tag number maybe
# If you have forked this project you can use <username>/Profile-Readme-WakaTime@master instead
- uses: avinal/Profile-Readme-WakaTime@master
with:
# WakaTime API key stored in secrets, do not directly paste it here
WAKATIME_API_KEY: ${{ secrets.WAKATIME_API_KEY }}
# Automatic github token
GITHUB_TOKEN: ${{ github.token }}
# Branch - newer GitHub repositories have "main" as default branch, change to main in that case, default is master
BRANCH: "master"
# Manual Commit messages - write your own messages here
COMMIT_MSG: "Automated Coding Activity Update :alien:"
```
6. Please wait till 12 AM UTC to run this workflow automatically. Or
you can force run it by going to the Actions tab. Or you can add the
following lines under `on:` to run with
every push. Search for 12 AM UTC to find the equivalent time in your
time zone.
```yaml
on:
push:
branches: [ master ]
schedule:
- cron: '1 0 * * *'
```
## My Coding Activity
![Avinal's GitHub stats](https://raw.githubusercontent.com/avinal/avinal/main/images/stat.svg)
static/content/posts/development/wsl1.md
+96
View File
@@ -0,0 +1,96 @@
---
title: Move WSL 2 Safely to another Drive
date: 2020-12-31 19:07
tags: [wsl, wsl2]
category: development
description: 'It is real pain when you have small SSD and Windows Subsystem for Linux
(WSL) is growing exponentially in size. There is no easy way to move the
WSL installation to another drive. Here in this blog I will discuss how
to tackle this problem with bite size steps.'
image: "/images/windows-wsl2.webp"
---
# Move WSL 2 Safely to another Drive
It is real pain when you have small SSD and Windows Subsystem for Linux
(WSL) is growing exponentially in size. There is no easy way to move the
WSL installation to another drive. Here in this blog I will discuss how
to tackle this problem with bite size steps.
1. Open a PowerShell or Command Prompt with *Admin* access. For this you can
use WinKey + X shortcut and select **Windows PowerShell(Admin)**.
2. Check if the WSL 2 installation you are planning to move is is
currently running/stopped.
```powershell
PS C:\\Users\\Avinal> wsl -l -v
PS C:\\Users\\Avinal>
NAME STATE VERSION
* Ubuntu Running 2
Kali Stopped 2
```
3. If its running then you must stop the particular WSL distribution.
(*Ubuntu* used as example)
```powershell
PS C:\\Users\\Avinal> wsl -t Ubuntu
```
4. Export to some folder. (Here exporting *Ubuntu* as *ubuntu-ex.tar*
to *Z:wsl2*)
```powershell
PS C:\\Users\\Avinal> wsl --export Ubuntu "Z:\\export\\ubuntu-ex.tar"
```
5. Unregister previous WSL installation
```powershell
PS C:\\Users\\Avinal> wsl --unregister Ubuntu
```
6. Create a new folder and import your WSL installation to that folder.
```powershell
PS C:\\Users\\Avinal> New-Item -Path "Z:\\wsl2" -ItemType Directory
Directory: Z:\\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 31-12-2020 21:03 wsl2
PS C:\\Users\\Avinal> wsl --import Ubuntu "Z:\\wsl2" "Z:\\export\\ubuntu-ex.tar"
```
7. Check after import is complete
```powershell
PS C:\\Users\\Avinal> wsl -l -v
PS C:\\Users\\Avinal>
NAME STATE VERSION
* Ubuntu Running 2
Kali Stopped 2
```
8. Mark one of your WSL distribution as *(default)*.
```powershell
PS C:\\Users\\Avinal> wsl -s Ubuntu
```
9. After exporting your default user will be set as
<i style="color:red">root</i> , to change it to your desired
username, run following command
```powershell
PS C:\\Users\\Avinal> ubuntu config --default-user user_name
```
10. Finally run `wsl` and you have successfully moved your WSL 2
installation to another drive.
## Attribution
- [Image](https://www.atwix.com/magento/magento-2-with-docker-for-windows-and-wsl-2/)