Gaston Jarju's Blog

  At my internship today, I continued working on the HTTPS GET request to the elasticsearch api. Having constructed my elasticsearch query in the Elasticsearch console, I just needed to use that query and pass it to my elasticsearch api call in my intelliJ code. The goal is to use the list of hostnames I queried from Big Query and get their metadata from the elasticsearch api. I used the olivere elasticsearch package because it is compatible with my http client. So too, I would just have to use the package’s functions to make the Get request to the api.  After writing the Get request function and passing the list of hostnames and pushing my code to Gitlab, it did not work on the first try. The data format is returned was in byte format and I would like to have it in Json readable format. Thus, I iterate through that array of byte data and convert them into json. However, the elasticsearch olivere package size can only take 10,000. This is a big issue  for my case because ...

  Ever heard the saying “Bad programmers worry about the code. Good programmers worry about data structures and their relationships?” Well, it is very much true in my scenario at my internship today. As a new Go programmer, I was much worried about writing the code and making it work as it should. I worried less about the code structure, but my manager is very worried because my project would be used in production and if someone else is going to work on it, they shouldn’t struggle to read and understand my code. Therefore, I was tasked to refactor my code. When I say refactoring, what I exactly mean is grouping my different functions into different directories and having different Go files. Since I have multiple logic for different API calls and data manipulation, I was required to arrange these logics and API calls into different directories and files.  Although both my Elasticsearch API  and BigQuery API  calls used the same http client, the API responses in both c...

  At my internship today, I was working on troubleshooting the 403 authorization error I had from my logs when I was testing my SSO Auth Token connection to the Elasticsearch API. Prior to today, I wrote a Golang code using the Golang Olivere search package to connect to the Elasticsearch API. Basically, I passed my http client and an elasticsearch method to check if an index exists on the cluster of the Elasticsearch API. Having confirmed whether the system user is authorized to make a call to the Elasticsearch API, I was left with very little hints on what I could be doing wrong. Then the question about, what’s next came?  Well, as usual, there are always smarter people around me in the team, so I asked around to see if someone had worked with Elasticsearch before. To my surprise, one of the VP on our team had prior knowledge, so he decided to help me out with my code. We did pair programming and he was the driver for the most part. I must say that he is very versed with Gol...

  At my internship today, I was tasked to write a GET Request Golang code to the Elasticsearch API. Having written a GET request previously for the BigQuery API, I thought it would be an easy and fast task to complete. However, it was the total opposite. With the Elasticsearch Rest API, I had to use a Golang package that could use my SSO Auth token to be able to have the permission to call the API, and to my surprise it is turning out to be more difficult than I previously envisioned. Because of security reasons, not all system users are granted permission to make an API call. I did not realize this until I made an API GET Request. From the look at my code, everything seems  correct, but when I pushed my code and took a look at my logs on the console, there was a 403 error code, which simply means unauthorize.  Even after the admin authorized my system user to make API calls, my code was not working still. I had to use a different approach. I tried testing my http client ...

  Today at my internship, I was tasked to create an elasticsearch query to generate different properties of a field. Elasticsearch is a Rest API search engine that makes searching for data very fast and straightforward. It involves using clusters and indexing to generate data. One can generate the data in two ways: by using the Elasticsearch CLI or writing an elasticsearch query. I chose to use the latter because I needed to write a Golang HTTPS request and pass that query in order to get the data that I specified in my query. Unlike how easy or rather simpler it was for me when I had to write the Big Query API query, it was a different experience today because elasticsearch query does not use SQL querying. It uses a DSL query, which simply refers to Domain Specific Language. The DAL is based on Json to define queries: key value pairing. Apart from the fact that I had no prior experience with DSL queries, I also did not have any experience with the Elasticsearch API. Therefore, I s...

  It sometimes feels like we are not good enough when our tries don’t work. Well at my internship today, I worked on resolving the json unmarshalling issue I was stuck with last Friday. After several back-and-forth consultations with my buddy and manager, I finally got a better understanding of how they wanted me to present the data. Basically, the data should be in a simple json array that could be manipulated easier. This is because as the json comes in key:pair format, I would need to extract all the keys and use those keys to send an HTTPS request to Elasticsearch Rest API.  Even though the approach was well defined and clear to me, I still struggled to implement it using the Golang Struct and Maps. One of the obstacles I had was because after unmarshalling the json data, the output was in a Golang interface type, which was not iterable like say a slice. A Golang slice is the equivalent  of a python array. Therefore, to overcome that barrier, I created an empty slice ...

  At my internship today, I continued unmarshalling the Json data I got back from Big Query API. Having tried replicating the data on the Go playground and using the Go unmarshall function yesterday, I did develop a better understanding of the json structure. I used Go’s struct to do the unmarshalling. Unlike the json data I got from my query, the documentation’s json data does not have any schema metadata and so I thought the same approach would not work. Little did I know that the schema data can be ignored when unmarshalling json data and that you can only concentrate on the part of the Json you want to unmarshall. When I consulted my buddy, he hopped on a zoom call with me and we did pair programming. I was the driver while he was the navigator, basically explaining what we were doing because he has more knowledge on Go and Json data manipulation. Working with him gave me more skills and confidence in what I was doing. He helped in not just using the Go struct, but also in tryi...

  Today at my internship, I was tasked at converting the json data format that my big query returns. My query from Big Query API returns a Json response body. Unlike the regular json array data I have seen previously, this json was different because first it has some schema metadata that I don’t need in my response body. The json data also is nested with each key in each map having the same key name, let say “v.” For me, this was the biggest challenge I have had with manipulating json data.   So how did it go for me? Well, I must say that it is really tedious. I first of all had to read more on manipulating nested json data and then use the Go programming language to do the conversion. After reading a couple of articles, I got an idea that I could use a Go package, unmarshall, to modify the json data. From the stack overflow solution I found, the unmarshall package was used for very simple json data: key: value pair map data type.    Thus, to mimic that approach was ...

Little did I know that a Kubernetes deployment project could have multiple clusters and namespaces. Perhaps, it was because I was all along working with one cluster, which also has one namespace since the genesis of my internship. Well, since I needed to make sure that my application has access to the Big Query API, I needed to use a different cluster and a namespace because my initial cluster does not have any big query data. Thus, my manager created a new cluster and namespace and gave me the names. Just like how I login the previous cluster, I also used the same command line to login to the new cluster. The only part of the command line I needed to change was the cluster name as well as the namespace name if I needed to use the pods in that namespace. In short, I now have two clusters: one is to access the Big query data and the only is used for my server application.  Unlike how easy it was to navigate and use one cluster, it is much more complex when dealing with two clusters ...

Today at my internship, I worked on activating my Big Query Console within the firm’s authentication and authorization and testing out some of the Queries on the data from the Big Query API. Big Query is a Google Cloud Platform, which is public as well. However, unlike how I can create an account to sign in to the Big Query Console, I had to use my company’s email to access the data housed in the Big Query. The Big Query is quite fascinating because it has the description of the data, the schemas and data types all outlined.  In addition, the console and executable run time is pretty fast and simple to use.  Like many databases and data, my data is in a relational database format. It is basically arranged in a set of rows and columns. This display eases the struggle of writing the query. One of the queries that my manager needed me to write was , “ A query that returns every host(distinct) and the number of system logs that contain a certain body massage.” Having worked with r...

  Having successfully deployed my Kurbenetes yesterday, today I started implementing the HTTPS GO Request Post. As my project will be using the Big Query API to query the data and post the result to my server, I was needed to make that client-server post request. Unlike the GET HTTPS Request, the POST request function accepts three parameters. The Go inbuilt Post function accepts these three parameters:  The URL address of the server, the content type of the body as a string, and the request body that is to be sent using the POST method of type. The documentation was pretty easy and simple to comprehend, so I had less problems implementing the POST functions. However, I had to restructure my code to meet my project goals. Unlike the documentation, I was not hard coding the data I needed to post to my server. Mine was a two-way procedure: I had to first write a query in the Swagger API, then pass that query as the request body in the Post Function. Then, that query will be post...

  I have not felt so satisfied achieving a milestone in a while like I did at my internship today. It feels so great to finally have your code do what you envisioned from the start. So how did this start? Well, a week ago, I started my Kubernetes deployment, Gitlab CI/CD integration and Go HTTPS GET Request. This deployment was meant to be the foundation to my HTTPS Go Get Request call. However, my CI/CD pipeline was failing and the debugging was rather difficult because I was unable to see my logs from the PUTTY CLI. And even after seeing the logs, it was still not giving much helpful information to where the error was coming from. What was the issue? Unfortunately or fortunately, the error was not from  the codebase I wrote. It was the company’s packages and libraries that were not up-to-date. As my buddy was helping me debug, he was able to point out that we were using the wrong version of the packages all this while.  Besides, I felt a great sigh of relief because I w...

  Today at my internship, I was assigned the issue of troubleshooting why my GET request Golang code is unable to generate the data from my http server. Since it was a combination of k8s deployment and Golang go, it was trickier and harder than I envisioned. The k8s was mainly used to orchestrate the objects that are needed to generate keytab from my pods. After several debugging: comparing and contrast the referred file I used as a source code coupled with looking up the errors on Stack flow (suggested me to delete my deployment and redeployed), I was still unable to fix the issue. Partly was because I did not understand what the error message was pointing at. Thus, I had to consult a member of the k8s team, who figured out that my deployment.yaml file was missing some configurations that needed to generate my key-tab. Simply, I needed the key-tab in order to be able to make the HTTP Get Request get my desired data.   Although the recommended solution from my consultation wor...