Gaston Jarju's Blog

  At my internship today, I gave my final project presentation to the firm’s Unix Global Team. Prior to my presentation, I did practice my slides a couple of times that morning to get over the nervousness and get more familiar with the flow of my presentation. So too, I was informed that the presentation was going to be held on zoom, so I made some speakers notes so that it would  help me remember what to say when I present. From my previous powerpoint presentation,. I learned that slides work best when I don't cram too much information - especially too much text onto them.  Although the initial plan from the team was to present on zoom, it got changed 30 minutes before the scheduled time. The reason why this change was made was because unlike otherwise anticipated, all the employees in the Jersey City office that work in my team came into the office today. Usually only 15-20% of the team comes in on the same day to work. Therefore, I had to mentally prepare myself becaus...

  At my internship today, I worked on finalizing my powerpoint presentation for my final Global Team presentation. For the powerpoint slides, I was supposed to follow the generic engineering slide template format. Therefore, I requested to have a look at the previous intern powerpoint slides so that I can have an idea on how to structure and outline my slides. After receiving the design template, I was surprised that it had little or no graphics. For me, the design template was not appealing and had less standardized colors and styles. As a result, I decided to incorporate more graphics and standardized colors to my slides.  Although I used the designed template I was handed, I tried to only include necessary information as well as used colors that contrast and complement each other. So too, I made sure that my powerpoint had a coherent flow of ideas. For instance, I made sure that with an introductory slide, an agenda slide, an overview of what my project is, the problem my p...

  At my internship today, I worked on cleaning my code and handing off my project to my team. Having worked on this project for the past nine weeks, it felt so satisfying and fulfilling that I had the opportunity to finally meet the project deliverables and hand off my project. My code was not the best written code because it was the first time programming in Golang for me; however, now that I looked at how much logic and functionalities my codebase does, it is truly intriguing.   So what does it really mean for me when I say clean my code. Well, as I mentioned earlier, this was my first time programming in Golang. Therefore, I did have a lot of print statements for debugging. I integrated these print statements because they helped me to easily find bugs as well as know the stage of my logic. Another code refactoring I had to do was removed commented out chunks of code. Since these code was no more needed in my codebase, I had to get rid of them so that it’d be more organized ...

At my internship today, I continued on completing the merging and outputting the final result on my server. Yesterday I was able to successfully compare the hostnames from the Elasticsearch API and Big Query API in the form of struct; however, I could not combine the two struct and transform the struct data type into a readable format, for instance Json. As a result, I spent my whole today resolving that issue. In simple terms, I have these two structs that I needed to merge after making the comparison and display the one merge data on my browser. To make sure the comparison I was making from yesterday was correct, I loop through the merge data and create a map. Maps are great choices because of its keys and values; however, it was not the best choice because my server is to display the final output into a json format and apparently using a struct makes that way more dynamic and easier. With a struct, I could easily convert the struct into a json format by unmarshalling it.  Althou...

  Today at my internship, I worked on merging the two data outputs I got from Elasticsearch API and Big query API calls I made. Basically, both these chunks of data have metadata of the hostnames and the hostnames itself. So too, they are in all byte format and thus I would have to change their format before being able to merge them together. As I have worked previously with converting bytes into a human readable form, I would think that it would be easier with me this time around. However, it was the total opposite because my case total was not only to convert the data format, but also to merge them.  So what did it pan out? I was successful in making it work, but it took me longer and more difficult to implement the logic. Prior to my implementation, I wrote a pseudocode to make it easier to follow. The logic I created was to unmarshall these two data into structs so that I can access the common hostnames in each struct, check if they are the same. And when they are alike, I...

  At my internship today, I continued working on resolving the query size limit that the olivere package poses. As I mentioned in one of my previous posts, the olivere elasticsearch package only returns 0-10 documents by default. This default of the size is an issue because my data pool is over hundred thousand of data. So too, the package does not allow a Bulk Https API request. Therefore, I had to come up with a solution to how to make multiple API calls and combine the chunks into one readable data structure. The idea is to break my hostname arrays into chunks and use each chunk to make the api call.  Although I had an idea of what I was supposed to do, I did not have any idea how to implement it in Go programming. Thus, I had to google it. After doing a couple of searches, I saw a similar logic on Stackoverflow, which gave me a better idea on how to implement the logic. As usual, I replicated a similar logic on the Go online IDE. Using the online IDE helps me troubleshoot ...

  At my internship today, I continued working on the HTTPS GET request to the elasticsearch api. Having constructed my elasticsearch query in the Elasticsearch console, I just needed to use that query and pass it to my elasticsearch api call in my intelliJ code. The goal is to use the list of hostnames I queried from Big Query and get their metadata from the elasticsearch api. I used the olivere elasticsearch package because it is compatible with my http client. So too, I would just have to use the package’s functions to make the Get request to the api.  After writing the Get request function and passing the list of hostnames and pushing my code to Gitlab, it did not work on the first try. The data format is returned was in byte format and I would like to have it in Json readable format. Thus, I iterate through that array of byte data and convert them into json. However, the elasticsearch olivere package size can only take 10,000. This is a big issue  for my case because ...

  Ever heard the saying “Bad programmers worry about the code. Good programmers worry about data structures and their relationships?” Well, it is very much true in my scenario at my internship today. As a new Go programmer, I was much worried about writing the code and making it work as it should. I worried less about the code structure, but my manager is very worried because my project would be used in production and if someone else is going to work on it, they shouldn’t struggle to read and understand my code. Therefore, I was tasked to refactor my code. When I say refactoring, what I exactly mean is grouping my different functions into different directories and having different Go files. Since I have multiple logic for different API calls and data manipulation, I was required to arrange these logics and API calls into different directories and files.  Although both my Elasticsearch API  and BigQuery API  calls used the same http client, the API responses in both c...

  At my internship today, I was working on troubleshooting the 403 authorization error I had from my logs when I was testing my SSO Auth Token connection to the Elasticsearch API. Prior to today, I wrote a Golang code using the Golang Olivere search package to connect to the Elasticsearch API. Basically, I passed my http client and an elasticsearch method to check if an index exists on the cluster of the Elasticsearch API. Having confirmed whether the system user is authorized to make a call to the Elasticsearch API, I was left with very little hints on what I could be doing wrong. Then the question about, what’s next came?  Well, as usual, there are always smarter people around me in the team, so I asked around to see if someone had worked with Elasticsearch before. To my surprise, one of the VP on our team had prior knowledge, so he decided to help me out with my code. We did pair programming and he was the driver for the most part. I must say that he is very versed with Gol...

  At my internship today, I was tasked to write a GET Request Golang code to the Elasticsearch API. Having written a GET request previously for the BigQuery API, I thought it would be an easy and fast task to complete. However, it was the total opposite. With the Elasticsearch Rest API, I had to use a Golang package that could use my SSO Auth token to be able to have the permission to call the API, and to my surprise it is turning out to be more difficult than I previously envisioned. Because of security reasons, not all system users are granted permission to make an API call. I did not realize this until I made an API GET Request. From the look at my code, everything seems  correct, but when I pushed my code and took a look at my logs on the console, there was a 403 error code, which simply means unauthorize.  Even after the admin authorized my system user to make API calls, my code was not working still. I had to use a different approach. I tried testing my http client ...

  Today at my internship, I was tasked to create an elasticsearch query to generate different properties of a field. Elasticsearch is a Rest API search engine that makes searching for data very fast and straightforward. It involves using clusters and indexing to generate data. One can generate the data in two ways: by using the Elasticsearch CLI or writing an elasticsearch query. I chose to use the latter because I needed to write a Golang HTTPS request and pass that query in order to get the data that I specified in my query. Unlike how easy or rather simpler it was for me when I had to write the Big Query API query, it was a different experience today because elasticsearch query does not use SQL querying. It uses a DSL query, which simply refers to Domain Specific Language. The DAL is based on Json to define queries: key value pairing. Apart from the fact that I had no prior experience with DSL queries, I also did not have any experience with the Elasticsearch API. Therefore, I s...

  It sometimes feels like we are not good enough when our tries don’t work. Well at my internship today, I worked on resolving the json unmarshalling issue I was stuck with last Friday. After several back-and-forth consultations with my buddy and manager, I finally got a better understanding of how they wanted me to present the data. Basically, the data should be in a simple json array that could be manipulated easier. This is because as the json comes in key:pair format, I would need to extract all the keys and use those keys to send an HTTPS request to Elasticsearch Rest API.  Even though the approach was well defined and clear to me, I still struggled to implement it using the Golang Struct and Maps. One of the obstacles I had was because after unmarshalling the json data, the output was in a Golang interface type, which was not iterable like say a slice. A Golang slice is the equivalent  of a python array. Therefore, to overcome that barrier, I created an empty slice ...

  At my internship today, I continued unmarshalling the Json data I got back from Big Query API. Having tried replicating the data on the Go playground and using the Go unmarshall function yesterday, I did develop a better understanding of the json structure. I used Go’s struct to do the unmarshalling. Unlike the json data I got from my query, the documentation’s json data does not have any schema metadata and so I thought the same approach would not work. Little did I know that the schema data can be ignored when unmarshalling json data and that you can only concentrate on the part of the Json you want to unmarshall. When I consulted my buddy, he hopped on a zoom call with me and we did pair programming. I was the driver while he was the navigator, basically explaining what we were doing because he has more knowledge on Go and Json data manipulation. Working with him gave me more skills and confidence in what I was doing. He helped in not just using the Go struct, but also in tryi...

  Today at my internship, I was tasked at converting the json data format that my big query returns. My query from Big Query API returns a Json response body. Unlike the regular json array data I have seen previously, this json was different because first it has some schema metadata that I don’t need in my response body. The json data also is nested with each key in each map having the same key name, let say “v.” For me, this was the biggest challenge I have had with manipulating json data.   So how did it go for me? Well, I must say that it is really tedious. I first of all had to read more on manipulating nested json data and then use the Go programming language to do the conversion. After reading a couple of articles, I got an idea that I could use a Go package, unmarshall, to modify the json data. From the stack overflow solution I found, the unmarshall package was used for very simple json data: key: value pair map data type.    Thus, to mimic that approach was ...

Little did I know that a Kubernetes deployment project could have multiple clusters and namespaces. Perhaps, it was because I was all along working with one cluster, which also has one namespace since the genesis of my internship. Well, since I needed to make sure that my application has access to the Big Query API, I needed to use a different cluster and a namespace because my initial cluster does not have any big query data. Thus, my manager created a new cluster and namespace and gave me the names. Just like how I login the previous cluster, I also used the same command line to login to the new cluster. The only part of the command line I needed to change was the cluster name as well as the namespace name if I needed to use the pods in that namespace. In short, I now have two clusters: one is to access the Big query data and the only is used for my server application.  Unlike how easy it was to navigate and use one cluster, it is much more complex when dealing with two clusters ...

Today at my internship, I worked on activating my Big Query Console within the firm’s authentication and authorization and testing out some of the Queries on the data from the Big Query API. Big Query is a Google Cloud Platform, which is public as well. However, unlike how I can create an account to sign in to the Big Query Console, I had to use my company’s email to access the data housed in the Big Query. The Big Query is quite fascinating because it has the description of the data, the schemas and data types all outlined.  In addition, the console and executable run time is pretty fast and simple to use.  Like many databases and data, my data is in a relational database format. It is basically arranged in a set of rows and columns. This display eases the struggle of writing the query. One of the queries that my manager needed me to write was , “ A query that returns every host(distinct) and the number of system logs that contain a certain body massage.” Having worked with r...

  Having successfully deployed my Kurbenetes yesterday, today I started implementing the HTTPS GO Request Post. As my project will be using the Big Query API to query the data and post the result to my server, I was needed to make that client-server post request. Unlike the GET HTTPS Request, the POST request function accepts three parameters. The Go inbuilt Post function accepts these three parameters:  The URL address of the server, the content type of the body as a string, and the request body that is to be sent using the POST method of type. The documentation was pretty easy and simple to comprehend, so I had less problems implementing the POST functions. However, I had to restructure my code to meet my project goals. Unlike the documentation, I was not hard coding the data I needed to post to my server. Mine was a two-way procedure: I had to first write a query in the Swagger API, then pass that query as the request body in the Post Function. Then, that query will be post...

  I have not felt so satisfied achieving a milestone in a while like I did at my internship today. It feels so great to finally have your code do what you envisioned from the start. So how did this start? Well, a week ago, I started my Kubernetes deployment, Gitlab CI/CD integration and Go HTTPS GET Request. This deployment was meant to be the foundation to my HTTPS Go Get Request call. However, my CI/CD pipeline was failing and the debugging was rather difficult because I was unable to see my logs from the PUTTY CLI. And even after seeing the logs, it was still not giving much helpful information to where the error was coming from. What was the issue? Unfortunately or fortunately, the error was not from  the codebase I wrote. It was the company’s packages and libraries that were not up-to-date. As my buddy was helping me debug, he was able to point out that we were using the wrong version of the packages all this while.  Besides, I felt a great sigh of relief because I w...

  Today at my internship, I was assigned the issue of troubleshooting why my GET request Golang code is unable to generate the data from my http server. Since it was a combination of k8s deployment and Golang go, it was trickier and harder than I envisioned. The k8s was mainly used to orchestrate the objects that are needed to generate keytab from my pods. After several debugging: comparing and contrast the referred file I used as a source code coupled with looking up the errors on Stack flow (suggested me to delete my deployment and redeployed), I was still unable to fix the issue. Partly was because I did not understand what the error message was pointing at. Thus, I had to consult a member of the k8s team, who figured out that my deployment.yaml file was missing some configurations that needed to generate my key-tab. Simply, I needed the key-tab in order to be able to make the HTTP Get Request get my desired data.   Although the recommended solution from my consultation wor...