diff --git a/static/content/posts/blogs/hey-tekton-results.md b/static/content/posts/blogs/hey-tekton-results.md index a49ba48..4e38511 100644 --- a/static/content/posts/blogs/hey-tekton-results.md +++ b/static/content/posts/blogs/hey-tekton-results.md @@ -1,2 +1,160 @@ --- ---- \ No newline at end of file +title: Tekton Results to the Rescue +date: 2023-03-16 20:47 +tags: [tekton, kubernetes, redhat, openshift, results] +category: blogs +image: "/images/tekton-results-wall.webp" +description: Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller. +--- + +# Tekton Results to the Rescue + +What do you do with your Tekton Pipelines once it finishes? Depending on if it +failed or passed, you may keep it to inspect the logs. For most of the users/organizations +the simplest step is to keep the completed TaskRuns/PipelineRuns object on the +cluster to retrieve the data later. + +Imagine having thousands of runs and although a single TaskRun object takes very +small space compared to the scale of a production cluster, but these little +things add up quickly, and soon your cluster will be burdened with objects that +probably no one will ever revisit. The organization/user is keeping them +just-in-case if they want to see the logs and other data later. + +Although in most of the cases there is a pruning policy that takes care of these +objects. But there are multiple problems with this approach. + +- This type of storage is not reliable and very difficult to query. +- If the scale is massive, this could lead to destabilization of the cluster. +- If you have a pruning policy, the completed objects are cleaned, and you lose all the associated data as well. + +So, what is the solution, how can you save your pipelines' data without having to keep them on cluster? + +## Introducing Tekton Results + +As mentioned on the [project repository](https://github.com/tektoncd/results): + +> Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller. This allows you to: +> +> - Provide custom Result metadata about your CI/CD workflows not available in the Tekton TaskRun/PipelineRun CRDs (for example: post-run actions) +> - Group related workloads together (e.g. bundle related TaskRuns and PipelineRuns into a single unit) +> - Make long-term result history independent of the Pipeline CRD controller, letting you free up etcd resources for Run execution. + +In short, Tekton results archives the run data (called results) and logs to an +external storage. Now you can safely prune completed TaskRuns/PipelineRuns and +save run data and logs for a later visit. Let us see how actually Tekton Results +works under the hood. + +### How Tekton Results Works? + +Tekton Results is composed of two main components. + +- A **watcher** to listen to creation/update of PipelineRuns or TaskRuns. +- An **API Server** to query the persistent storage for data. + +In addition to that, Tekton Results needs a working database connection that can +be a persistent storage on the same cluster or hosted externally such as RDS. +If no external storage is attached, logs are also stored on a persistent storage +on the cluster, you may use a S3 (or compatible) storage solution for that. + +The lifecycle of a *result* is as below: + +1. The first step is to create a Tekton PipelineRun or TaskRun. +2. The Watcher listens for any changes in the TaskRun or PipelineRun. +3. On change, Watcher updates (or creates) a corresponding `Record` or `Result` using the Results API. + Watcher adds annotations to the TaskRuns or PipelineRuns with proper identifiers. Watcher uses + these annotations to decide if the `Result` has been created/updated/finished or not. +4. You can now query the Results data using the API. If the run state is incomplete yet, the response + from the API will indicate that as well via the status flag. +5. Once the TaskRun/PipelineRun has been completed, you can safely prune the resource object. + +## Installing Tekton Results + +Installing Tekton Results is easy. You can use Kubernetes or OpenShift cluster, for this particular +demonstration, I will be using a Kind cluster and a local database. + +### Prerequisites + +- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) for a local Kubernetes cluster. +- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) +- [curl](https://curl.se/download.html) for quering the API. +- [OpenSSL](https://www.openssl.org/source/) for generating certificates. + +### Let's start + +1. Create a Kind Cluster + + ```sh + kind create cluster --name tekton-results + kind export kubeconfig --name tekton-results + ``` + +2. [Tekton Pipelines]() must be installed on the cluster. You can install it using the command below. + + ```sh + kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml + ``` + +3. Generate a database root password and store as a Kubernetes Secret. If you are using an external + database, prove the credential for the same. Here is a bare minimum requirement as YAML. + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: tekton-results-postgres + namespace: tekton-pipelines + type: Opaque + data: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: + ``` + + You can directly use the command line as well: + + ```sh + kubectl create secret generic tekton-results-postgres --namespace="tekton-pipelines" --from-literal=POSTGRES_USER=postgres --from-literal=POSTGRES_PASSWORD=$(openssl rand -base64 20) + ``` + +4. Generate a cert/key pair. You may use any cert management software to generate this. You can even + use cluster generated certs. + + ```sh + openssl req -x509 \ + -newkey rsa:4096 \ + -keyout key.pem \ + -out cert.pem \ + -days 365 \ + -nodes \ + -subj "/CN=tekton-results-api-service.tekton-pipelines.svc.cluster.local" \ + -addext "subjectAltName = DNS:tekton-results-api-service.tekton-pipelines.svc.cluster.local" + ``` + +5. Create another TLS Kubernetes Secret with the name `tekon-results-tls` to store the cert/key pair. + + ```sh + kubectl create secret tls -n tekton-pipelines tekton-results-tls \ + --cert=cert.pem \ + --key=key.pem + ``` + +6. Install Tekton Results + + ```sh + kubectl apply -f https://storage.googleapis.com/tekton-releases/results/latest/release.yaml + ``` + +7. You can check the status of the deployments using the below command. Do not worry + if some deployments show `CrashLoopBackOff`. Wait for some time, and + they should all be running. + + ```sh + kubectl get pods -n tekton-pipelines --watch + ``` + +Once all deployments are ready, we can start creating some TaskRuns/PipelineRuns. In the next part +of this blog, I will explain how to retrieve data from Tekton Results. Happy Reading. + +## References + +- [Tekton Results](https://github.com/tektoncd/results) +- [Tekton Pipelines](https://github.com/tektoncd/pipelines) diff --git a/static/content/posts/blogs/hrt-interview-1.md b/static/content/posts/blogs/hrt-interview-1.md index 1d1163c..d5cbb5a 100644 --- a/static/content/posts/blogs/hrt-interview-1.md +++ b/static/content/posts/blogs/hrt-interview-1.md @@ -2,7 +2,7 @@ title: HRT (Hudson River Trading) Systems Internship Interview Experience date: 2021-01-04 21:47 tags: [HRT, hudsonrivertrading, interview, internship] -category: blog +category: blogs image: "/images/hrt-singapore.webp" --- diff --git a/static/content/posts/development/wakatime.md b/static/content/posts/development/wakatime-readme.md similarity index 100% rename from static/content/posts/development/wakatime.md rename to static/content/posts/development/wakatime-readme.md diff --git a/static/content/posts/posts.json b/static/content/posts/posts.json index afbffd4..861352d 100644 --- a/static/content/posts/posts.json +++ b/static/content/posts/posts.json @@ -1 +1 @@ -[{"title":"Developing Minimal Tekton Server","slug":"lovely-dangerous-things-redhat","image":"/images/development.webp","description":"This blog is a descreptive account of the development of Minimal Tekton Server. This is highly technical in nature, so please make sure that you have sufficient knowledge about Golang, Docker, Kubernetes and TektonCD.","date":"2022-02-27 20:47","category":"development"},{"title":"I am loving it! RedHat","slug":"i-am-loving-it-redhat","image":"/images/fedora-wall.webp","description":"I made it to the Red Hat as a DevTools intern. This post is about onboarding and how I prepared myself for working on the actual project.","date":"2022-02-25 20:47","category":"development"},{"title":"GSoC'21 Final Evaluation Report","slug":"final-evaluation","image":"/images/gsoc-wall.webp","description":"This is the final report of my Google Summer of Code 2021 at The FOSSology Project. One of the major improvements over the previous build system is faster build times. CMake generates parallel build-enabled configurations for all generators.","date":"2021-08-19 23:07","category":"gsoc"},{"title":"GSoC'21 Coding Week 10","slug":"meeting-11","image":"/images/tech-wallpaper-12.webp","description":"This week I implemented CMake testing configuration and fixed most of the tests. As of now all but 5 tests are working fine.","date":"2021-08-14 22:47","category":"gsoc"},{"title":"GSoC'21 Coding Week 9","slug":"meeting-10","image":"/images/tech-wallpaper-11.webp","description":"This week I worked on CMake testing configuration. Most of the time was spent understanding the previous testing architecture.","date":"2021-08-06 22:47","category":"gsoc"},{"title":"GSoC'21 Coding Week 8","slug":"meeting-9","image":"/images/tech-wallpaper-10.webp","description":"This week I implemented CMake packaging configuration for FOSSology. The new configuration fixes issue with previous packaging configurations. It also retains the component wise installation features.","date":"2021-07-30 22:47","category":"gsoc"},{"title":"GSoC'21 Coding Week 7","slug":"meeting-8","image":"/images/tech-wallpaper-9.webp","description":"This week I implemented CMake packaging configuration for FOSSology. There were two meetings in this week and this report covers both of them.","date":"2021-07-23 22:22","category":"gsoc"},{"title":"GSoC'21 First Evaluation Report","slug":"first-evaluation","image":"/images/tech-wallpaper.webp","description":"In the first phase of GSoC 2021 at The FOSSology Project, I have completed the desired milestone. As of now, FOSSology can be installed completely via CMake and most of the components are working fine in initial testing.","date":"2021-07-14 12:29","category":"gsoc"},{"title":"GSoC'21 Coding Week 5","slug":"meeting-7","image":"/images/tech-wallpaper-8.webp","description":"This week was dedicated to perfecting CMake Installation Configuration. The installation was tested and bugs were discussed.","date":"2021-07-09 22:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 4.2","slug":"meeting-6","image":"/images/tech-wallpaper-7.webp","description":"In this eighth meeting questions related to post install generation were asked. This was a short meeting.","date":"2021-07-02 22:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 4.1","slug":"meeting-5","image":"/images/tech-wallpaper-6.webp","description":"In this seventh meeting question related to installing the FOSSology were discussed.","date":"2021-06-29 23:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 3","slug":"meeting-4","image":"/images/tech-wallpaper-5.webp","description":"In this fifth meeting, question related to versioning and obtaining commit hash were discussed, this was a short meeting.","date":"2021-06-22 23:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 2","slug":"meeting-3","image":"/images/tech-wallpaper-4.webp","description":"In this fourth meeting, a lot of questions were discussed related to the existing build system and what things we have to drop or modify.","date":"2021-06-18 23:30","category":"gsoc"},{"title":"GSoC'21 Coding Week 1","slug":"meeting-2","image":"/images/tech-wallpaper-3.webp","description":"In this third meeting, I demoed the working build system, currently building executables and libraries, a lot of queries were resolved about writing version files and attaching commits and hashes to the build.","date":"2021-06-11 23:30","category":"gsoc"},{"title":"GSoC'21 Community Bonding Week 1","slug":"meeting-1","image":"/images/tech-wallpaper-2.webp","description":"In this second meeting points over default Makefiles were discussed. Ninja can be used as an alternative for Makefiles.","date":"2021-06-04 22:30","category":"gsoc"},{"title":"GSoC'21 Community Bonding Week 0","slug":"meeting-0","image":"/images/tech-wallpaper-1.webp","description":"This meeting is the first of the recurring weekly GSoC project meetings. In this meeting the current status of progress according to the proposal was discussed and some topics related to current build system based on Make and the new build system based on CMake.","date":"2021-05-28 21:30","category":"gsoc"},{"title":"Hello CMake","slug":"cmake-basics","image":"/images/cmake-office.webp","description":"CMake stands for Cross-platform Make. Normally, a build tool like Make will parse a configuration file (Makefile) that contains all the commands required to build an artifact based on the source files and other resources inside the project.","date":"2021-05-24 23:56","category":"development"},{"title":"How I implemented WakaTime embeddable Coding Graph GHA?","slug":"wakatime","image":"/images/waka.webp","description":"If you use WakaTime to track your coding activity. You can add that to your README as a bar graph or embed it in your blog/portfolio. Just add this action to any of your repositories and there you have it.","date":"2021-02-02 21:47","category":"development"},{"title":"HRT (Hudson River Trading) Systems Internship Interview Experience","slug":"hrt-interview-1","image":"/images/hrt-singapore.webp","description":"uestions were clear and of medium level. But they were designed in such a way that you must know the basics before you could attempt. Also, they expected a clear and concise approach.","date":"2021-01-04 21:47","category":"blogs"},{"title":"Move WSL 2 Safely to another Drive","slug":"wsl1","image":"/images/windows-wsl2.webp","description":"It is real pain when you have small SSD and Windows Subsystem for Linux (WSL) is growing exponentially in size. There is no easy way to move the WSL installation to another drive. Here in this blog I will discuss how to tackle this problem with bite size steps.","date":"2020-12-31 19:07","category":"development"},{"title":"Create the VLC User Documentation for one Mobile Port(Android)","slug":"gsod2020-report","image":"/images/day-of-cone.webp","description":"The project was to Create the VLC User Documentation for Android Mobile Port which was previously hosted on VLC wiki pages. The major portion of this was to start everything from scratch including chapter separation, section organization.","date":"2020-12-01 23:47","category":"blogs"},{"title":"प्रेम रतन धन पायो","slug":"for-sunshine","image":"/images/fedora-night.webp","description":"प्रकृति की सुंदरता और कलाकारी हिमालय की कण-कण में झलकती है। प्रकृति ने प्रेम को भी हिमालय के जितना ही विशाल और अलौकिक बनाया है । ये एक अलग चर्चा का विषय है कि हिमालय पहले आया या प्रेम। मैं तो प्रेम के पक्ष में हूँ । वो हर अणु-परमाणु जिन्होंने इतने बड़ा पहाड़ खड़ा किया वो सब आपस में प्रेम से बंधे हुए हैं। ये पृथ्वी, सूर्य, चंद्रमा, आकाश-गंगा इत्यादि सब प्रेम से बंधे हुए हैं","date":"2019-09-21 15:47","category":"articles"},{"title":"The Big Red Ants","slug":"big-red-ants","image":"/images/ants.webp","description":"As I observed them making and reparing their nests, I concluded that they are laborious and intellectual. They create their nests bybinding two or more leaves (maybe up to 500) together. They stich the leaves using a stinky white substance either excreted by themselves or from trees.","date":"2012-02-27 22:47","category":"articles"}] \ No newline at end of file +[{"title":"Tekton Results to the Rescue","date":"2023-03-16 20:47","slug":"hey-tekton-results","category":"blogs","image":"/images/tekton-results-wall.webp","description":"Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller."},{"title":"Developing Minimal Tekton Server","slug":"lovely-dangerous-things-redhat","image":"/images/development.webp","description":"This blog is a descreptive account of the development of Minimal Tekton Server. This is highly technical in nature, so please make sure that you have sufficient knowledge about Golang, Docker, Kubernetes and TektonCD.","date":"2022-02-27 20:47","category":"development"},{"title":"I am loving it! RedHat","slug":"i-am-loving-it-redhat","image":"/images/fedora-wall.webp","description":"I made it to the Red Hat as a DevTools intern. This post is about onboarding and how I prepared myself for working on the actual project.","date":"2022-02-25 20:47","category":"development"},{"title":"GSoC'21 Final Evaluation Report","slug":"final-evaluation","image":"/images/gsoc-wall.webp","description":"This is the final report of my Google Summer of Code 2021 at The FOSSology Project. One of the major improvements over the previous build system is faster build times. CMake generates parallel build-enabled configurations for all generators.","date":"2021-08-19 23:07","category":"gsoc"},{"title":"GSoC'21 Coding Week 10","slug":"meeting-11","image":"/images/tech-wallpaper-12.webp","description":"This week I implemented CMake testing configuration and fixed most of the tests. As of now all but 5 tests are working fine.","date":"2021-08-14 22:47","category":"gsoc"},{"title":"GSoC'21 Coding Week 9","slug":"meeting-10","image":"/images/tech-wallpaper-11.webp","description":"This week I worked on CMake testing configuration. Most of the time was spent understanding the previous testing architecture.","date":"2021-08-06 22:47","category":"gsoc"},{"title":"GSoC'21 Coding Week 8","slug":"meeting-9","image":"/images/tech-wallpaper-10.webp","description":"This week I implemented CMake packaging configuration for FOSSology. The new configuration fixes issue with previous packaging configurations. It also retains the component wise installation features.","date":"2021-07-30 22:47","category":"gsoc"},{"title":"GSoC'21 Coding Week 7","slug":"meeting-8","image":"/images/tech-wallpaper-9.webp","description":"This week I implemented CMake packaging configuration for FOSSology. There were two meetings in this week and this report covers both of them.","date":"2021-07-23 22:22","category":"gsoc"},{"title":"GSoC'21 First Evaluation Report","slug":"first-evaluation","image":"/images/tech-wallpaper.webp","description":"In the first phase of GSoC 2021 at The FOSSology Project, I have completed the desired milestone. As of now, FOSSology can be installed completely via CMake and most of the components are working fine in initial testing.","date":"2021-07-14 12:29","category":"gsoc"},{"title":"GSoC'21 Coding Week 5","slug":"meeting-7","image":"/images/tech-wallpaper-8.webp","description":"This week was dedicated to perfecting CMake Installation Configuration. The installation was tested and bugs were discussed.","date":"2021-07-09 22:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 4.2","slug":"meeting-6","image":"/images/tech-wallpaper-7.webp","description":"In this eighth meeting questions related to post install generation were asked. This was a short meeting.","date":"2021-07-02 22:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 4.1","slug":"meeting-5","image":"/images/tech-wallpaper-6.webp","description":"In this seventh meeting question related to installing the FOSSology were discussed.","date":"2021-06-29 23:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 3","slug":"meeting-4","image":"/images/tech-wallpaper-5.webp","description":"In this fifth meeting, question related to versioning and obtaining commit hash were discussed, this was a short meeting.","date":"2021-06-22 23:22","category":"gsoc"},{"title":"GSoC'21 Coding Week 2","slug":"meeting-3","image":"/images/tech-wallpaper-4.webp","description":"In this fourth meeting, a lot of questions were discussed related to the existing build system and what things we have to drop or modify.","date":"2021-06-18 23:30","category":"gsoc"},{"title":"GSoC'21 Coding Week 1","slug":"meeting-2","image":"/images/tech-wallpaper-3.webp","description":"In this third meeting, I demoed the working build system, currently building executables and libraries, a lot of queries were resolved about writing version files and attaching commits and hashes to the build.","date":"2021-06-11 23:30","category":"gsoc"},{"title":"GSoC'21 Community Bonding Week 1","slug":"meeting-1","image":"/images/tech-wallpaper-2.webp","description":"In this second meeting points over default Makefiles were discussed. Ninja can be used as an alternative for Makefiles.","date":"2021-06-04 22:30","category":"gsoc"},{"title":"GSoC'21 Community Bonding Week 0","slug":"meeting-0","image":"/images/tech-wallpaper-1.webp","description":"This meeting is the first of the recurring weekly GSoC project meetings. In this meeting the current status of progress according to the proposal was discussed and some topics related to current build system based on Make and the new build system based on CMake.","date":"2021-05-28 21:30","category":"gsoc"},{"title":"Hello CMake","slug":"cmake-basics","image":"/images/cmake-office.webp","description":"CMake stands for Cross-platform Make. Normally, a build tool like Make will parse a configuration file (Makefile) that contains all the commands required to build an artifact based on the source files and other resources inside the project.","date":"2021-05-24 23:56","category":"development"},{"title":"How I implemented WakaTime embeddable Coding Graph GHA?","slug":"wakatime","image":"/images/waka.webp","description":"If you use WakaTime to track your coding activity. You can add that to your README as a bar graph or embed it in your blog/portfolio. Just add this action to any of your repositories and there you have it.","date":"2021-02-02 21:47","category":"development"},{"title":"HRT (Hudson River Trading) Systems Internship Interview Experience","slug":"hrt-interview-1","image":"/images/hrt-singapore.webp","description":"Questions were clear and of medium level. But they were designed in such a way that you must know the basics before you could attempt. Also, they expected a clear and concise approach.","date":"2021-01-04 21:47","category":"blogs"},{"title":"Move WSL 2 Safely to another Drive","slug":"wsl1","image":"/images/windows-wsl2.webp","description":"It is real pain when you have small SSD and Windows Subsystem for Linux (WSL) is growing exponentially in size. There is no easy way to move the WSL installation to another drive. Here in this blog I will discuss how to tackle this problem with bite size steps.","date":"2020-12-31 19:07","category":"development"},{"title":"Create the VLC User Documentation for one Mobile Port(Android)","slug":"gsod2020-report","image":"/images/day-of-cone.webp","description":"The project was to Create the VLC User Documentation for Android Mobile Port which was previously hosted on VLC wiki pages. The major portion of this was to start everything from scratch including chapter separation, section organization.","date":"2020-12-01 23:47","category":"blogs"},{"title":"प्रेम रतन धन पायो","slug":"for-sunshine","image":"/images/fedora-night.webp","description":"प्रकृति की सुंदरता और कलाकारी हिमालय की कण-कण में झलकती है। प्रकृति ने प्रेम को भी हिमालय के जितना ही विशाल और अलौकिक बनाया है । ये एक अलग चर्चा का विषय है कि हिमालय पहले आया या प्रेम। मैं तो प्रेम के पक्ष में हूँ । वो हर अणु-परमाणु जिन्होंने इतने बड़ा पहाड़ खड़ा किया वो सब आपस में प्रेम से बंधे हुए हैं। ये पृथ्वी, सूर्य, चंद्रमा, आकाश-गंगा इत्यादि सब प्रेम से बंधे हुए हैं","date":"2019-09-21 15:47","category":"articles"},{"title":"The Big Red Ants","slug":"big-red-ants","image":"/images/ants.webp","description":"As I observed them making and reparing their nests, I concluded that they are laborious and intellectual. They create their nests bybinding two or more leaves (maybe up to 500) together. They stich the leaves using a stinky white substance either excreted by themselves or from trees.","date":"2012-02-27 22:47","category":"articles"}] \ No newline at end of file diff --git a/static/images/tekton-results-wall.webp b/static/images/tekton-results-wall.webp new file mode 100644 index 0000000..2a7cd79 Binary files /dev/null and b/static/images/tekton-results-wall.webp differ