WEEK 9: ENTRY 1

 At my internship today, I continued working on resolving the query size limit that the olivere package poses. As I mentioned in one of my previous posts, the olivere elasticsearch package only returns 0-10 documents by default. This default of the size is an issue because my data pool is over hundred thousand of data. So too, the package does not allow a Bulk Https API request. Therefore, I had to come up with a solution to how to make multiple API calls and combine the chunks into one readable data structure. The idea is to break my hostname arrays into chunks and use each chunk to make the api call. 


Although I had an idea of what I was supposed to do, I did not have any idea how to implement it in Go programming. Thus, I had to google it. After doing a couple of searches, I saw a similar logic on Stackoverflow, which gave me a better idea on how to implement the logic. As usual, I replicated a similar logic on the Go online IDE. Using the online IDE helps me troubleshoot faster and see the bigger picture of my logic. After replicating the logic, it did work. However, unlike the array of integer data types that I found in the solution, my data type is an array of string. So I had to make it work for my array of strings. Having worked with arrays of strings previously, it was easy for me to make this change. Basically, I needed to change the function’s parameters to take in an array of strings and return a string data type. Unlike how python takes parameters and return variables without specifying their data types, Go programming requires you to do those type declarations. The changes I made did resolve the issue and I was able to pass the same logic to my Go project’s file. 


After adding the same logic to my project, it did return a data type error. I had to check my logs to see where this was occurring. It occurred that it was the same mistake I made last week, which was failing to pass my elasticsearch api call output to the Get function. I felt very bad because it was an easy fix and did not require my manager to spot it. I would have love to resolve the whole issue by myself and less assistance from my manager. In my next attempts to resolve the issue, I plan to do it solely by myself with less supervision because this way, I would be confident that I’m making more progress and actually remembering the things I learned previously.


Comments

Post a Comment

Popular posts from this blog

WEEK 4: ENTRY 3

  Today at my internship, I was assigned the issue of troubleshooting why my GET request Golang code is unable to generate the data from my http server. Since it was a combination of k8s deployment and Golang go, it was trickier and harder than I envisioned. The k8s was mainly used to orchestrate the objects that are needed to generate keytab from my pods. After several debugging: comparing and contrast the referred file I used as a source code coupled with looking up the errors on Stack flow (suggested me to delete my deployment and redeployed), I was still unable to fix the issue. Partly was because I did not understand what the error message was pointing at. Thus, I had to consult a member of the k8s team, who figured out that my deployment.yaml file was missing some configurations that needed to generate my key-tab. Simply, I needed the key-tab in order to be able to make the HTTP Get Request get my desired data.   Although the recommended solution from my consultation wor...
Read more

WEEK 6: ENTRY 3

  At my internship today, I continued unmarshalling the Json data I got back from Big Query API. Having tried replicating the data on the Go playground and using the Go unmarshall function yesterday, I did develop a better understanding of the json structure. I used Go’s struct to do the unmarshalling. Unlike the json data I got from my query, the documentation’s json data does not have any schema metadata and so I thought the same approach would not work. Little did I know that the schema data can be ignored when unmarshalling json data and that you can only concentrate on the part of the Json you want to unmarshall. When I consulted my buddy, he hopped on a zoom call with me and we did pair programming. I was the driver while he was the navigator, basically explaining what we were doing because he has more knowledge on Go and Json data manipulation. Working with him gave me more skills and confidence in what I was doing. He helped in not just using the Go struct, but also in tryi...
Read more

WEEK 7: ENTRY 1

  It sometimes feels like we are not good enough when our tries don’t work. Well at my internship today, I worked on resolving the json unmarshalling issue I was stuck with last Friday. After several back-and-forth consultations with my buddy and manager, I finally got a better understanding of how they wanted me to present the data. Basically, the data should be in a simple json array that could be manipulated easier. This is because as the json comes in key:pair format, I would need to extract all the keys and use those keys to send an HTTPS request to Elasticsearch Rest API.  Even though the approach was well defined and clear to me, I still struggled to implement it using the Golang Struct and Maps. One of the obstacles I had was because after unmarshalling the json data, the output was in a Golang interface type, which was not iterable like say a slice. A Golang slice is the equivalent  of a python array. Therefore, to overcome that barrier, I created an empty slice ...
Read more