Splunk announced layoffs impacting 7 percent of staff ahead of its deal to be acquired by Cisco for $28 billion.

Splunk Cuts 7% Of Staff; ‘Not A Result’ Of Cisco Deal, CEO Says

In the latest market close, Splunk (SPLK) reached $147.16, with a -0.03% movement compared to the previous day. The stock trailed the S&P 500, which registered a daily gain of 0.65%. On the other hand,

Splunk (SPLK) Stock Slides as Market Rises: Facts to Know Before You Trade

In the latest trading session, Splunk (SPLK) closed at $147.49, marking a -0.12% move from the previous day. The stock exceeded the S&P 500, which registered a loss of 1.43% for the day. Meanwhile, the Dow experienced a drop of 0.

Splunk (SPLK) Stock Moves -0.12%: What You Should Know

Principal Analyst, Mobility & VR, Anshel Sag analyzes the Snapdragon 8 Gen 3 and Snapdragon X Elite SoCs and predicts their impact on AI-driven mobile devices and PCs.

AI Dominates Qualcomm Snapdragon Summit With New Snapdragon Products

Goodson shares his thoughts on Atom Computing's new second-generation 1,225-qubit neutral atom quantum computer.

Atom Computing Announces Record-Breaking 1,225-Qubit Quantum Computer

Splunk was upgraded to the Zacks Rank #1 list on August 29, 2023. The Zacks Rank is a unique stock-rating model that helps you take advantage of earnings estimate revision trends and provides a way to get into stocks highly sought after by institutional investors.

The Zacks Rank Explained: How to Find Strong Buy Computer and Technology Stocks

It doesn't matter if you're a growth, value, income, or momentum-focused investor -- building a successful investment portfolio takes skill, research, and a little bit of luck. But what's the best way to find the right combination of stocks?

In this article, I will delve into whether artificial intelligence (AI), particularly the emergence of tools like generative AI (GenAI), will replace classical software outsourcing as we know it.

AI vs software outsourcing: An opportunity or a threat?

From Microsoft, Google parent Alphabet, Atlassian and Salesforce-owned Slack, Dopbox works with a slew of tech leaders. That has the relative strength line for DBX stock floating rising up to a 52-week high.

Cloud Stock Floats Near Buy Point In Stormy Market

Clearly, putting AI on the edge is a beneficial path forward for enterprises looking to lower their power bills and environmental impacts.

How moving AI to the edge can help solve the data center energy crisis

SPLUNK INC

splunkinc

What happens when you take Splunk, a wireless smart meter and an api interface? Splunking your entire home’s power main, without touching a single power wire! This allows a whole house view of all the power being drawn in real time. The only requirements are raspberry pi on a network, a splunk instance somewhere and a usb adapter plugged into the PI. Hold on, hold on, whats a smart meter? Recently the major power companies have been going around and replacing the old power meters on everybody’s house with a new fancy smart meter. This is essentially just a zigbee enabled power meter than allows power vans to roll down the street and collect all the readings quickly, without having a meter reader go house to house on foot. This page or the page for the a similar provider can be useful when identifying what kind of meter is installed. Then identify the usb dongle that is compatible to the smart meter, most sites have a list of those devices. Then order it up and get it registered via power company. A simple process however it requires paperwork and some lead time. The device that will be used here is the Rainforest EMU-2 Energy Monitoring Unit. / https://www.comed.com/SmartEnergy/SmartMeterSmartGrid/Pages/ForYourHome.aspx https://www.amazon.com/gp/product/B00BGDPRAI Enough of the prework time to dive in. The bulk of the work will be done with a python script that reads from the usb adapter, which is recognized as a serial port on the pi. Then we will install the python script as a service on the pi, which will allow it to run on boot and be easily restarted, and monitored. The python script: There are two requirements here pyRaven and Splunk_hec_handler, those can be found here: The splunk hec handler can also be found in pypi, however at the time of writing the version on pypi was broken and would not pip install, go after the source instead, then do the https://github.com/vavarachen/splunk_hec_handler https://github.com/nonspecialist/pyraven python setup.py install Great we have the requirements and the script. The script is pretty straight forward it connects to the serial port, it will listen to the serial firehose and parse when the data flows across. This is where the usb device is just terrible it has options to query the api but then there is a 4 second lag, during which time the same data will spit out automatically and then again from the query. It is really easier to sit there and listen to the firehose. Regardless the pyraven library does all the work and we are plugging in to it to grab the json object and pass it to a splunkhec function. The splunkhec function receives the data and appends the needed wrapper and posts the event. The posting of the event is then handled by the splunkhec library because why reinvent the wheel and deal with rest posts when we can use a library to do it. What this gives us is a nice json object being loaded into splunk. After we examine the data we can see that a simple splunk search, will give us the graph below. host=gate | timechart avg(event.raw_demand) as Load span=1m The service: In order for the pi to run this script automagically a service template is used and then installed. The service is assuming the the emu.py from above is going to be store in folder. /home/pi/ Alright the next step is to make sure the python script is executable and then install the service template and get it running. That can all be done with the snippet below. Thats all there is to it, the service is running the python script and outputting a whole bunch of nice data to play with. Once the data is acquired then the possibilities are endless for monitoring and reporting out of splunk for high usage or anomalous usage. ***BONUS: (JQ is required for parsing) In order to get the data in to splunk a HTTP collector token is needed. Use curl to generate a token: LINKS: _Splunk indexes and makes searchable data from any app, server or network device in real time including logs, config…_dev.splunk.com Use the CLI to administer HTTP Event Collector | Splunk _This repository allows you to access your Rainforest Automation Emu2 through it's serial interface …_github.com rainforestautomation/Emu-Serial-API _A Python Logging Handler for Splunk HTTP Event Collector (HEC). - vavarachen/splunk_hec_handler_github.com vavarachen/splunk_hec_handler _Interface with a Rainforest Automation RAVEn USB device - nonspecialist/pyraven_github.com nonspecialist/pyraven

Splunking Power Mains

Unlocking IaC Part 3: Your First Terraform Deployment!

IaC পার্ট 3 আনলক করা: আপনার প্রথম টেরাফর্ম স্থাপনা!

IaC Teil 3 freischalten: Ihre erste Terraform-Bereitstellung!

Desbloqueo de IaC Parte 3: ¡Su primera implementación de Terraform!

Déverrouillage d'IaC, partie 3 : votre premier déploiement Terraform !

IaC भाग 3 को अनलॉक करना: आपका पहला टेराफ़ॉर्म परिनियोजन!

IaC のロックを解除するパート 3: 初めての Terraform デプロイメント!

IaC 파트 3 잠금 해제: 첫 번째 Terraform 배포!

Desbloqueando IaC Parte 3: Sua primeira implantação do Terraform!

Открываем IaC, часть 3: ваше первое развертывание Terraform!

IaC Bölüm 3'ün Kilidini Açma: İlk Terraform Kurulumunuz!

Mở khóa IaC Phần 3: Triển khai Terraform đầu tiên của bạn!

解锁 IaC 第 3 部分：您的首次 Terraform 部署！

These are some AI-generated project names, courtesy of Namelix . Our naming story I write about developer innovation and new technologies for a VC firm called Project A. Every now and then, we develop projects that we’d like to open source. Coming up with names for these projects is always a challenge and each team has a different approach. I’d like to describe how we named a couple of our projects and what we learned along the way. One of these projects was initiated by our data engineering team and another by our back-end team. Despite having different use cases, both of these projects have a similar theme — they enable developers to efficiently process data from different sources and pipe it to different systems. Naming is hard but important Coming up with a name is never an easy decision. Babies, bands, top-secret projects, new products, and companies — someone, somewhere has to name them. Naming an open source project isn’t quite as “high-stakes” as naming a baby but it ain’t a walk in the park either. You can give an internal development project whatever crappy name you want, but if there’s a chance you’re going to go public with it, there are several reasons why you should put some effort into a decent name (besides simply avoiding ridicule). “A good name can help a company or product become successful, of course, but it can also help the lowliest code library find an audience, help formalize an informal process, and propel ideas about the world toward becoming talking points throughout it.” – Greg Leppert and Willem Van Lancker, Onym Developers can learn a lot from marketing pros Admittedly, we didn’t ask our marketing team for any tips about naming. They’re busy folks and we didn’t want to bother them with a highly technical project that we were going to share for free. For open source projects, you probably won’t need to worry about traditional marketing anyway — at least not when starting off. However, earlier in my career, I had collaborated with marketing teams on product naming and got to know some of the experts that they trusted. I started reading what these experts had written about naming and branding. Some of their ideas and concerns are still highly relevant for open source software. For example, I love this quote: “ “ We should lower our sights when it comes to naming. Because company names, like pretty much everything in life, fall on a bell curve. There are a few fantastic names, a whole bunch of okay-to-good names and a few truly awful names. The problem is, the surest way to end up with an awful name is by aiming for a fantastic one. — Doug Kessler, Velocity Partners It addresses how much time you can potentially waste trying to come up with the perfect name, and foreshadows some of the problems that we personally encountered. The first problem is… Who to involve in the naming process Whatever you do, you should keep the group small — I would recommend no more than 5 people. Any larger and you’re going end up with costly review cycles and discussions. Also, decide the group in advance and stick to it — gradually involving more and more people can undo any previous work and bring you back to square one. Or you could also leave the entire responsibility up to a senior member of your team. The creator of Maria DB, Max DB, and MySQL simply named those projects after his kids: “Maria”, “Max” and “My”. However, this strategy didn’t go so well for us. The backend project was first conceived a couple of years ago, by our CTO and a few of his colleagues. They were joking around and thought it would be fun to name the future project “Wayne” (no one had any children named Wayne). Now, I don’t want to offend any people with that name, but we felt it had certain connotations that we wanted to avoid. Not really the association we were going for — Wayne on YouTube (Image: YouTube) In all fairness to our CTO, “Wayne” was just a working title. He quickly needed a name so that he could discuss the thing that we wanted the team to build…without calling it “the thing”. However, most of us agreed that we needed a better name. Avoid a free-for-all naming frenzy Many development teams have flat hierarchies and try to involve everyone in big decisions. So, of course, we sought input from everyone in the team. Unfortunately, this strategy can lead to a lot of…shall we say…“noise”. Think of what happens whenever an organization decides to let the internet suggest a name for a product or project. As the New York Times wrote in 2016: happens. Nerds will be nerds. Boaty McBoatface And so it was, that names like “Crunchy McCrunchFace” and “BusinessObject McBusinessObjectFace” were some of the first suggestions when we tried to name our own projects (full disclosure, the second one was mine). People are also going to suggest names from popular culture such as “Gargamel”, “Tyrion”, and “Optimus Prime” — but good luck if you want people to find your “Tyrion” project in Google. Of course, it’s fun to do some free association but it probably wouldn’t do any harm to skip this part. Automating the brainstorming process Marketing teams have all sorts of complex methodologies that they like to use for this. For example, , Doug Kessler describes how they plotted words on scales such as “evocative vs descriptive”, “emotional vs rational” and “playful vs enterprisey”. They also arranged words that sent “the right signals” into a sphere of associations. in his article on rebranding If this stuff floats your boat, then by all means, give it a try — but I’m guessing you’d rather be coding than pinning names to an ideas board. And luckily, some of these processes have been automated by AI. While we were trying to figure out how to come up with a name, our CTO discovered an interesting tool called . According to their website, Namelix “uses artificial intelligence to create a short, brandable business name”. Namelix We weren’t after a business name but were game to give it a try. Namelix wants you to enter a few keywords about your business, so I entered the most generic words that related to our tool “backend” “database” “software”. I opted for a medium length name and chose a “name style”. There are 8 styles to choose from and I opted for “Misspellings” (like Lyft and Flickr). Some of the suggestions were baffling: “Scraink”, “Boonry”, or “Treesus”? If you need to come up with a name for your emo rap trio, the “Misspellings” option might work for you. But there was no way we were going to name our project “treesus”. I tried again with the “Compound Word” option (like “FedEx” and “Instagram”). Those suggestions were even odder: “Nannystate”, “Spyware”, or “Failapp”? In all fairness to Namelix, I did cherry-pick the funniest examples, but nonetheless, I couldn’t find anything that I liked and I didn’t want to spend hours tinkering with the settings. However, Namelix did help with clarifying the of words that we should consider. To see what I mean, have a look at their “name style” options: types Incidentally, our data engineering team had structured their naming process around on similar “naming styles”. Which leads me to the next point: Agree on your naming styles before you start brainstorming Our data engineering team had people propose names in categories such as “Italian words”, “Swahili words”, “Minerals”, “Characters from Roman Mythology”, “Plant Names” and “verbs”. If you agree on the categories that you want to use first, you can avoid the task of filtering through a whole bunch of random junk. Focus on Verbs The “verbs” category is particularly important — you want to build a list of verbs that describe what your project Even if none of them are suitable for a name, verbs are especially useful for triggering inspiration. does. To start off, our data engineering came up with simple verb-object pairs that described their project. Here are some examples: Import data Build dashboards Display data Transform data Extract, transform, load (ETL) data For each of those principal “actions”, they brainstormed further verb “associations”. For example, here are some examples of verbs that they associated with the “ETL” action. organize manage sort merge gather fold They then translated these verbs into Swahili. Swahili seems to be a favored language when it comes to naming projects. I suspect that Swahili words appeal to English and German speakers because they have a melodic texture and are pleasing to vocalize (maybe this has something to do with the ). bouba/kiki effect Again, here are some of the ETL verbs with their Swahili translations. organize: kuandaa extract: dondoo manage: kusimamia sort: aina merge: kuunganisha gather: kukusanya fold: mara The name that the data engineering team liked best was the last one: ‘mara’ — it was short, simple and easy to pronounce. Sometimes a good backstory provides the best option Our backend team had more of an intuitive approach. For them, it was more important that their name had a good backstory or anecdote that they could share. Our lead backend developer tried to think of metaphors for his team’s project. Then he hit upon a childhood memory — he remembered how much he loved making herbariums at school. The Herbarium — our lead developer’s childhood passion. He felt that the “herbarium” was a good metaphor for his team’s project. The project was an application for defining business objects independent of any database or tool. Different systems could then pull data whatever structure they require. For our lead backend developer, each business object was kind of like a pressed plant. Each plant has its own shape and attributes, but it gets flattened into a consistent form so that all of these plant samples can be stored and compared together in one place. He shared this idea with the team and everyone liked the metaphor. Some people felt like “herbarium” was a bit of a mouthful, so they shortened it to the more diminutive “Herbie”. But their naming quest wasn’t over yet… Do the Google Test Before you make a final decision on the name, make sure you find out what shows up on the search results page when you search for that word online. Branding professionals do this so that they can find out how stiff the competition is for the search results rankings. This isn’t a huge priority for most open source projects — you’re not trying to build a brand, you just want to share your project. However, you don’t want to find out too late that your project name has some unexpected connotations or associations (For example, Ikea had to deal with this problem when they ). expanded into Thailand Incidentally, our data engineering team was not immediately aware that “mara” also refers to an odd-looking mammal. That was until our Chief Data Officer went to the zoo with his family and spotted this sign: Fun fact: Here in Germany, there’s a very logical system for naming animals (as illustrated by this German animal names flowchart ) so in this neck of the woods, the “mara” is also called “Pampas Hare” (Pampashase). Fortunately, the Mara is kind of cute and it made for a great logo. Not everyone comes off so lucky, however. The point being, it’s better to be safe than sorry — even for a small open source project. What’s more important is searching on Github. Do the Github test Try checking the following URL patterns to see what comes up for your chosen name: https://github.com/<name> https://<name>.github.io We like to store our projects under their own organizations in Github so that they’re independent entities from the main “Project A” organization. These special projects consisted of several repos and we didn’t want them to get mixed up with all the other Project A repos. That’s why it was important for our data engineering team to get the account and for our backend team to get . https://github.com/mara https://github.com/herbie However, these short snappy names are often taken, but if you’re lucky, the name might belong to an inactive account. You can find out by reviewing the account activity. If there’s little to no activity, you might be able to claim the name according to . This was indeed the case for both the “mara” and “herbie” accounts. After contacting GitHub support, both names were made available for our projects. Github’s username policy Unfortunately for the backend team, they hadn’t paid enough attention to the general search results in Github. After everyone agreed on the name and set up the repos, people started discovering other “herbies” such as a German project and another from the University of Washington. Maybe “treesus” wasn’t such a bad idea after all? flat-file CMS small project While a double up is not ideal, it’s also not as critical as having a duplicate brand name. As long as the projects are in separate niches of the open source ecosystem, you can probably get away with it for a while. However, if another project with the same name becomes astronomically popular, your project risks drowning in a sea of content about its more well-known namesake. Check relevant top-level domain names This is not really a “must-have” but if you think that your project might gain traction, you could also check top-level domains that are popular with open source projects, such as “.org” or “.io”. Just be aware that some domains can come with a hefty price tag. At the time of writing, the domain “mara.io” was being sold for $1,999 — which definitely wasn’t an option for our data engineering team. Try out some usage examples As a final precaution, try to use the name in at least 5 sentences and make sure that there are no contexts where it sounds weird. For example, have a look at these sentences containing the term “Splunk” — the name of a popular application for analyzing machine-generated big data. “…to your new data, run the demo script…” “…you have the data but how will it be ?…” “…see our tutorial on Perforce Data…” “…check to see if knows about your script” “…both apps are enabled on this instance..” splunk splunked splunking Splunk Splunk Splunk is definitely a memorable name and a strong brand, but for British English speakers, there’s something disconcerting about using the word “Splunk” as a verb — it sounds somehow vulgar. A list of usage examples can help uncover such linguistic idiosyncrasies and it can also give you ideas on rules for correct usage. Further Resources Of course, I couldn’t finish a blog post about names without mentioning the indispensable naming resource “ ”. It’s a fantastic website which links to curated content about every aspect of the naming process, including many more tools that can help you with brainstorming. Onym I wish we had found out about this resource a lot earlier — we could have saved ourselves a lot of time!. But hopefully, these tips help you avoid some of the naming pitfalls that we encountered. Also, if you have any interesting stories about how you came up with a project name, let us know! A Disclosure Note On The Author and Project A Merlin blogs about developer innovation and new technologies at Project A — a venture capital investor focusing on early-stage startups. provides operational support to its portfolio companies, including developer expertise. As part of the IT team, he covers the highlights of Project A’s work helping startups to evolve into thriving success stories. Project A

How To Name Your Next Open Source Project

Ladies and gentlemen, I officially can say that the time has come. It has been an amazing 10 weeks of this crazy rollercoaster of a coding bootcamp. Full of emotions and enriches of friendships, I never expected this day to just be around the corner when I was watching another cohort experiencing the same adventure that I am about to undergo. If you are at Makers Academy, you’ve guessed it. My time has come for . Why is this moment so grand you ask for a student at Makers? Well, apart from the fact you are considered a senior… final project week Me trying to be as calm and confident as possible. You’ve finished all your foundation training of what the course has given you, and set out to practice. This is probably the last thing you’ll do at Makers This is the project, where in two weeks, you give it your best shot. This is where, when everyone heightens their senses and takes it up a notch because this is the moment to be proud off and prove it Although that sounds quite dramatic, I have to say it’s exciting and fun. It’s as exciting or entertaining as watching that scene in KungFu Panda where he’s completed all his training, and finally he is tested by his Shifu on using kung fu to be able to eat a bao. If you haven’t seen it, I recommend it. Before I start documenting this going to be unforgettable journey, . I think it’s the reason why I started blogging to begin with, to track all the lessons I have learn and continue to learn. I will say that I am writing this to document and remember my learning progress If that didn’t make sense, I guess what I mean is to l and . This is something I am practicing diligently because it stressed me out before when I didn’t and I realised I could be stopping the magic unknown moments that can come along the way. et the story unfold itself have no expectation on it Thus, I encourage you to do so. Am I excited about it? Yes. Am I going to give it my best shot? Yes. Am I ready for it? No, and who is? What is a final project (briefly): The final project is a two week schedule, where you are delegated into a team to produce a coding project base on your skills that you learn. Dependant on the team, you can choose what idea to work on and what languages to use. However, this project is different from the practice ones you’ve done because you have to consider things that you didn’t have to. Possible Things such as: The user flow for the project The goals of the project The way the project is going to be structured Once your final project is done, you have to present your project in front of coaches, peers, possible hiring managers, family and friends. The audience is a mix of support and a comedy crowd. Final projects finally give you the freedom of control to determine all of these based on what one learnt in the ten weeks. It’s a Makers Dream but it’s for sure easy to get your knickers in a twist. How the journey started Where does a cohort begin when it comes to dealing with final projects? A brainstorm session about ideas! On a Friday morning, the cohort gathered together to place either the topics, languages, product ideas or any other ideas on the board that a team could make. The ideas did not have to be great. It could be a prompt to something that didn’t have to be on the board. What this exercise did, for the coach (shoutout to Kay) and for us was to see what topics the cohort were thinking about it. Interesting stuff came up. Things like cryptocurrency, machine- , hacking a program that is meant to be unhackable, Tinder for animals. learning Yeah, Great stuff. Kay then grouped them into categories and based on a dot vote system, eliminated the categories that were not most of interest. The categories for our cohort were the following: Machine Learning Tricky Product Product Learning a new Language Technical Challenge From then onwards, it was base on our individual’s fate to choose our teams base on our interests. It definitely was a nerve-wrecking wait given that it all depended on one coach to determine our fate. However, she did not let me down. I don’t know whether it’s the fact she’s simply too good or have been having cameras everywhere of how this cohort has interacted, but she definitely placed me in a Dream team. The journey of TeamVEM: You’re probably wondering what is VEM, but that is not important. That will come later. Team VEM, including myself, is a team with the goal of creating a product. Whether it is a web app or mobile app, it has to be something somewhat solid. Lewis, Tom’s hand & Eva ❤ Our first group chat started late in the evening after a refreshed week of being alone doing tech tests. What we decided to do was to go through all the ideas that were categorised, and see which ideas we were not interested in. An elimination process was good to do first, given that it focused our group’s interest on the ideas. Once, we had a group of ideas, we then categorised them into themes of what we all were interested in. Our result was that we had one big concept following four topics that sparked our interests. Our overall concept was wanting to make something “interactive + educational” Our categories were Messaging/Social, Service, Tracker/Mapping and Software Service Although some topics interlinked or integrated with each other, it was good have bubbles of topics around so we knew the general interests of the group. We then met, Kay, the guider of our idea session, to specify our ideas further. It was a fulfilling experience to go through, as I felt I was experiencing what it was like to have an open yet professional discussion on what holes we saw in ideas and what potential we saw in others. However, we came to a conclusion that still needed a specification. “We want to build something interactive and educational with maps” Like the abstract form of the jelly shape surrounding all these ideas, this never started to wiggle in my mind, and it continued to wiggle throughout the rest of the evening. The fact that we weren’t solid on something yet, or knowing what product we were actually making spun the wheels in my mind. Since it had to make us think of our approach in this way: “How could I educate the user on a topic?” Which led to more questions such as: Why would you want to be educated on something? A long life lesson once reminded me that “Whys” are questions that go in circles if you can’t break down the question to other questions to hows or whats. Hence, I stopped immediately or needed the force or ping pong to stop my brain from spinning its gears. As it still wiggles my nerves, the positive part is that it can be still formed into something exciting or has the potential to still be nurtured. This reminded me of a script from the earlier mentioned movie, Kung Fu Panda that was the remedy to these thoughts. Master Oogway: “I cannot make it blossom when it suits me, nor make it bear fruit for me Shifu: But there are things you can control. I can control where the fruit will fall and where the seed is planted. That is no illusion! Master Oogway: No matter what you do with the seed, it will be a peach tree. You may wish for an apple tree, you may wish for an orange tree but it will always remain a peach tree. Shifu: But a peach tree cannot defeat Tai lung! Master Oogway: Maybe it can. If you are willing to guide it, to nurture it and believe in it. The lesson learnt here is to believe and nurture in the process and experience. To also nurture the idea, and believe in the idea no matter what flourishing thing it turns into. My move (research for the weekend): I believe it is an advantage if you come from a different background and enter in tech world. This is because you have more to share about the different places or experiences you come from. Hence, coming from a design background, I immediately knew that an inspiration for ideas of wanting to make “something educational, interactive with maps” could have the potential for data visualisation. Coming from a design background, looking at infographics designed around this topic would spark ideas of what innovative and creative topics people have chosen to do with maps, or in other ways. It also showed what potential data was out there for us to grab! Plus, it’s always a refreshing time for your eyes where you can look at printed material for immediate inspiration. Similar to design industry, the coding industry is a world that extracts from many different industries in the world because it’s a -based kind of industry. tool Conclusion of the start: Yeah, it’s great. It’s exciting. However, words of advice from having gone through the wave of emotions of the first day: If you are feeling nervous with the people you are working, don’t leap to conclusions about them just yet. They are growing just like you are, and they’ve probably learnt some major lessons along the same journey as you have. If you are feeling scared of being the weakest link, stop. It’s a team effort, and you will bring something more to the table which you don’t know yet. Don’t underestimate your ability or capability before the journey begins. If you are feeling a negative note on the whole situation, stop. Again, if you expect things already, you are blocking out the magical opportunistic things that could happen throughout the weeks. You mind is setting directions without looking at the whole map of where the project could take you. And if you are feeling all of these feelings, it’s okay to feel them. Like any big ‘milestone’ or ‘thing’, nerves are bound to come with something that is exciting. And with that, I sign off starting a new journey with #vems Just don’t let them linger on, for the sake of your sanity and your team’s vibe. Fun Fact of the Day: Other useful things to do when you start a final project, if you are stuck on what to do, is to talk about the following: What everyone’s goal is to get out from the project? What styles of standups and retros people are comfortable with doing. What people consider as important for the project? (Things like testing, wanting to set up the testing framework first, splunking to make sure everyone’s on the same page. What roles do people want to do? Yes, it may be awkward to put out there but By doing this, you are benefitting yourself and your team members to be more in sync with each other, before you even start. it’s necessary.

Episode 40: The final trials begin [Final Project Weeks]

Organizations now recognize the significant role of integrating security in the software development lifecycle. DevSecOps is a powerful approach that has emerged to achieve this goal. By incorporating security practices at the earliest stages of development, DevSecOps helps organizations identify and reduce vulnerabilities more effectively, minimize the risk of security breaches, and build more secure applications. This article will discuss the fundamental principles of DevSecOps and provide key steps for securing your organization’s CI/CD Pipeline. DevSecOps Principles : Organizational principles It is important to involve Security teams right from the beginning of a project to make sure that security requirements are given serious thought and attention early on. This step can prevent the discovery of security problems later in the development phase, which could be more expensive and time-consuming to fix. Involve Security Teams Early: In DevSecOps, it is important for Development, Security, and Operations teams to work together. By promoting a culture of shared responsibility for security, organizations can ensure that all team members play a part in creating secure applications and in identifying and solving security issues faster. Collaboration and Transparency: DevSecOps incorporates security practices early in the development process, also known as "shifting left." It helps organizations to detect vulnerabilities in the codebase before they escalate into expensive and time-consuming issues. Shift-Left: DevSecOps advocates for the inclusion of security procedures throughout the entire software development process, ensuring continuous security. This involves implementing automated security testing, constant monitoring, and regular security evaluations. Continuous Security: The concept of security chaos engineering involves deliberately introducing security-related problems and disruptions into a system to verify its ability to withstand them and reveal weaknesses. Security Chaos Engineering: By replicating actual security incidents, organizations can obtain significant insights into their system's performance during challenging conditions. This enables them to take proactive measures to address vulnerabilities and enhance their overall security position. Security Awareness/Training principles: To make security a top priority of your organization, provide your team members with security training and resources, encourage collaboration, and recognize that everyone is responsible for security. By doing this, you can cultivate a culture that prioritizes and integrates security throughout your organization. Foster a Security Culture: Developers should be encouraged to adopt secure coding practices like input validation, output encoding, and proper error handling. These practices will help mitigate security vulnerabilities in the code and reduce the risk of potential attacks. Adopt Secure Coding Practices: Gamification techniques can be adopted by organizations to encourage a security-focused culture and educate team members on security best practices. Gamification of Security Training: By transforming security training into an enjoyable and competitive experience, engagement and knowledge retention can be enhanced, ultimately promoting better security practices throughout the organization. Technical principles: Automating security processes can bring various benefits to organizations. These include increased efficiency, reduced human error, and the consistent application of security best practices. Automation: To maintain security during the development process, it is important to regularly run security checks and tests within the continuous integration and continuous deployment pipeline. This will ensure that security concerns are constantly evaluated and addressed. Integrate Security into the CI/CD Pipeline: Use tools like static application security testing (SAST) and dynamic application security testing (DAST) to check for weaknesses and potential security threats in your codebase. These automated security testing tools should be added to your organization's CI/CD pipeline to provide real-time security feedback. Implement Automated Security Testing: Securing the CI/CD Pipeline: Key Steps and Solutions To secure your CI/CD pipeline, protecting the Build environment is the first essential step. Only authorized personnel should access Build servers and infrastructure. To prevent unauthorized access, implement robust authentication methods, such as multi-factor authentication (MFA) with hardware keys. Additionally, keep the Build environment up to date with the latest security patches, and configure it according to the best industry practices. Secure the Build Environment: To safeguard your code repository, implement robust access controls that permit only the necessary access to authorized personnel, and monitor it frequently to identify any suspicious activities. Also, keep reviewing and updating user permissions and use encryption to protect sensitive data stored in the repository. Secure Code Repository: Hardcoded credentials, such as passwords or secret keys within the code, is a serious security risk as they can provide unauthorized access if the code is exposed. Hardcoded Credentials Scanner Solutions: Open-source solutions: TruffleHog - https://github.com/trufflesecurity/trufflehog - GitLeaks https://github.com/gitleaks/gitleaks - GitSecrets https://github.com/awslabs/git-secrets Enterprise solutions: GitGuardian - https://www.gitguardian.com/ SpectralOps - https://spectralops.io/ Bridgecrew - https://bridgecrew.io/secrets-scanning/ To automatically detect any potential security weaknesses in your code during the build process, integrate SAST tools into the CI pipeline. These tools can help you find problems such as insecure coding practices, input validation errors, and potential code injection vulnerabilities. Static Application Security Testing (SAST): Open-source solutions: SonarQube - https://www.sonarsource.com/products/sonarqube/ Semgrep CLI - https://github.com/returntocorp/semgrep Enterprise solutions: Veracode - https://www.veracode.com/products/binary-static-analysis-sast Checkmarx - https://checkmarx.com/cxsast-source-code-scanning/ - Coverity https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html To ensure the security of your application, you can use SCA tools to track third-party libraries and open-source components that you use. SCA tools can check for potential security risks such as outdated components, known vulnerabilities, and licensing issues. Software Composition Analysis (SCA): Open-source solutions: OWASP Dependency-Check - https://owasp.org/www-project-dependency-check/ - Nancy (GoLang) https://gonancy.dev/ - RetireJS (JavaScript) https://retirejs.github.io/retire.js/ - Safety (Python) https://github.com/pyupio/safety Enterprise solutions: - Black Duck Hub https://www.synopsys.com/software-integrity/security-testing/software-composition-analysis.html Mend - https://www.mend.io/sca/ Snyk - https://snyk.io/ In case your software uses containers, make sure to follow container security best practices, which include implementing minimal base images, scanning for vulnerabilities, and enforcing strict access controls. Additionally, it would be best if you used specialized container security tools, like container vulnerability scanners and runtime security monitoring solutions. Container Security: Open-source solutions: Trivy - https://github.com/aquasecurity/trivy - Grype https://github.com/anchore/grype - Clair https://github.com/quay/clair Enterprise solutions: Aqua Security - https://www.aquasec.com/products/software-supply-chain-security/ Sysdig Secure - https://sysdig.com/products/secure/ Snyk - https://snyk.io/product/container-vulnerability-management/ Incorporate security checks into your use of IAC tools like Terraform, CloudFormation, or Ansible. This is to ensure that the configuration of your infrastructure is indeed secure and free from potential security misconfiguration. Infrastructure as Code (IaC) Security: Open-source solutions: Checkov - https://github.com/bridgecrewio/checkov Tfsec - https://github.com/aquasecurity/tfsec KICS - https://kics.io/ Enterprise solutions: Bridgecrew - https://bridgecrew.io/infrastructure-as-code-security/ Prisma Cloud - https://www.paloaltonetworks.com/prisma/cloud/infrastructure-as-code-security - Snyk https://snyk.io/product/infrastructure-as-code-security/ To test your application in a real-time environment, add DAST tools to your CI/CD pipeline. These tools can uncover vulnerabilities that may only become visible when the application is executing, such as configuration issues, authentication problems, and access control concerns. Automated DAST tools should be used for thorough security testing. Dynamic Application Security Testing (DAST): Open-source solutions: OWASP Zed Attack Proxy (ZAP) - https://www.zaproxy.org/ - Nikto https://github.com/sullo/nikto - GoLismero https://github.com/golismero/golismero Enterprise solutions: Fortify WebInspect - https://www.microfocus.com/en-us/cyberres/application-security/webinspect Rapid7 InsightAppSec - https://www.rapid7.com/products/insightappsec/ - Checkmarx https://checkmarx.com/checkmarx-dast/ To enhance the security of your application and infrastructure, it is essential to continuously track and analyze their behavior. This can be achieved by implementing solutions for constant monitoring and logging. By doing so, you can quickly identify and respond to any potential security incidents that might occur. Continuous Monitoring and Logging: To facilitate this process, tools like SIEM (Security Information and Event Management) systems, intrusion detection systems, and log analyzers can be used to detect and alert you on security events. Open-source solution: ELK Stack (Elasticsearch, Logstash, and Kibana) - https://www.elastic.co/ Enterprise solutions: Splunk - https://www.splunk.com/en_us/products/enterprise-security.html Datadog - https://www.datadoghq.com/ To maintain the security and adherence of your CI/CD pipeline to industry standards and regulations, it is essential to conduct regular security audits and assessments. This includes examining the code, testing for potential vulnerabilities, and assessing the infrastructure. By implementing these measures, you can ensure that your pipeline is secure and compliant. Regular Security Audits and Assessments: Open-source solutions: OpenVAS - https://github.com/greenbone/ - BurpSuite Community https://portswigger.net/burp/communitydownload Enterprise solutions: Tenable Nessus - https://www.tenable.com/products/nessus QualysGuard - https://www.qualys.com/cloud-platform/ - BurpSuite Professional https://portswigger.net/burp/pro To be well-prepared for any security breach or incident, it is essential to create a clear incident response plan that lists the necessary steps to take. To ensure the safety of your application, your CI/CD pipeline must have the ability to roll back any deployments, patch any vulnerabilities, and immediately redeploy secure versions of your application. Incident Response Planning: Open-source solution: TheHive Project - https://thehive-project.org/ Enterprise solutions: IBM Resilient - https://www.ibm.com/products/qradar-soar Cortex XSOAR - https://www.paloaltonetworks.com/cortex/cortex-xsoar - D3Security SOAR https://d3security.com/enterprise-soar/ is a beneficial tool for DevSecOps teams. It's an open-source vulnerability management tool created by the OWASP community. Vulnerability Management and Reporting: DefectDojo This tool helps manage security vulnerabilities in the development process. One central space for all the information regarding defects is provided by DefectDojo. It makes it easier for teams to generate detailed reports and keep track of their progress over time. DefectDojo promotes clear communication about vulnerabilities, which ensures that security issues are resolved efficiently and promptly. As you can see, it is crucial to secure the CI/CD pipeline to implement a successful DevSecOps process. To achieve this, organizations need to incorporate security checks and tools throughout the pipeline. By doing so, they can identify vulnerabilities in advance and address them before they can cause security breaches. This helps to ensure that their applications are secure and comply with regulations. With the steps and practices outlined in this article, you can establish a robust and secure CI/CD pipeline for delivering software quickly and safely.

DevSecOps Principles and Key Steps for Securing the CI/CD Pipeline

SAS is an s a powerful analytics platform that allows users to perform complex data analysis and modeling tasks. To ensure high levels of performance and scalability, SAS offers a three-tier architecture that separates the different components of the system onto separate servers. This architecture allows for the metadata, data processing, and user interface layers to be installed on different servers for improved performance and scalability. In this article, we will discuss the steps to install a SAS three-tier architecture on an . AWS environment Provisioning EC2 Instances The first step in setting up a SAS three-tier architecture on AWS is to provision three EC2 instances. One instance will be used for the SAS Metadata Server, the second for the SAS Workspace Server, and the third for the SAS Stored Process Server. The size of the instances will depend on the size of your data and the expected load on your system. Here is an example Terraform code to provision three EC2 instances: provider "aws" { region = "us-east-1" } resource "aws_instance" "metadata_server" { ami           = "ami-0c94855ba95c71c99" instance_type = "t2.micro" tags = { Name = "metadata-server" } } resource "aws_instance" "workspace_server" { ami           = "ami-0c94855ba95c71c99" instance_type = "t2.micro" tags = { Name = "workspace-server" } } resource "aws_instance" "stored_process_server" { ami           = "ami-0c94855ba95c71c99" instance_type = "t2.micro" tags = { Name = "stored-process-server" } } Above code, we are using the AWS provider to set the region to us-east-1. We then define three aws_instance resources: metadata_server, workspace_server, and stored_process_server. Each instance has a specified Amazon Machine Image (AMI) and instance type and is tagged with a name for easy identification. Terraform Note that this is a basic example and you may need to modify the code to suit your specific requirements, such as adding additional configuration options, security settings, or networking components. Installing and Configuring the Required Software After provisioning the EC2 instances, the necessary software components need to be installed and configured on each of the instances. This includes installing the operating system, , database software, and SAS software. The specific versions of the software will depend on the version of SAS you are using and the requirements of your organization. Java Runtime Environment (JRE) Here is an example Ansible playbook for installing Java Runtime Environment (JRE), database software, and SAS software (SAS Preferred to installing GUI): name: Install JRE, Database Software, and SAS Software hosts: all become: true vars: java_version: "1.8.0" database_type: "oracle" database_version: "12c" sas_version: "9.4" tasks: name: Install JRE yum: name: "java-{{ java_version }}-openjdk" state: present name: Install {{ database_type }} Database Software yum: name: "{{ database_type }}{{ database_version }}" state: present name: Download SAS Software get_url: url: "https://example.com/sas/{{ sas_version }}/sas_{{ sas_version }}.tar.gz" dest: "/tmp/sas_{{ sas_version }}.tar.gz" name: Extract SAS Software unarchive: src: "/tmp/sas_{{ sas_version }}.tar.gz" dest: "/opt/sas" remote_src: true owner: sas group: sas name: Install SAS Software shell: "/opt/sas/setup.sh -deploy" args: chdir: "/opt/sas/install_data" Setting up the Metadata Server The SAS Metadata Server manages all metadata and user credentials for the SAS environment. Install and configure the SAS Metadata Server on the first EC2 instance. During the installation, you will need to specify the database and the location of the metadata repository. After the installation is complete, configure the metadata server using the SAS Management Console. Setting up the Workspace Server The SAS Workspace Server provides a connection point for SAS clients to execute SAS code and interact with data. Install and configure the SAS Workspace Server on the second EC2 instance. During the installation, you will need to specify the metadata server and the location of the workspace server data. After the installation is complete, configure the workspace server using the SAS Management Console. Setting up the Stored Process Server The SAS Stored Process Server provides a connection point for external applications to execute SAS programs and retrieve results. Install and configure the SAS Stored Process Server on the third EC2 instance. During the installation, you will need to specify the metadata server and the location of the stored process server data. After the installation is complete, configure the stored process server using the SAS Management Console. Configuring Network Settings After installing and configuring all the necessary components, network settings need to be configured to ensure that the three EC2 instances can communicate with each other. Firewall settings, security groups, and network interfaces need to be configured to allow communication between the three servers. It is important to follow best practices for securing your SAS environment on AWS. Testing the Installation Once all the components have been installed and configured, the SAS installation needs to be tested. Connect to the SAS Workspace Server using a SAS client application and run SAS programs on the Stored Process Server to verify that the results are returned correctly. By following the steps outlined in this article, you can install and configure a SAS three-tier architecture on AWS and achieve high levels of performance and scalability for your SAS environment. It is important to follow best practices for securing your SAS environment on AWS and to seek the assistance of a SAS expert if needed. Monitoring  a SAS Three-Tier Architecture using CloudWatch, and Splunk: CloudWatch Monitoring Amazon is a monitoring service for AWS resources and the applications running on them. You can use CloudWatch to collect and track metrics, collect and monitor log files, and set alarms. To monitor a SAS Three-Tier Architecture using , you can use the following steps: CloudWatch CloudWatch Enable CloudWatch logging for each EC2 instance running the SAS Metadata Server, SAS Workspace Server, and SAS Stored Process Server. This will allow CloudWatch to collect log data from each instance. Create CloudWatch alarms to monitor specific metrics for each instance. For example, you can create alarms to monitor CPU usage, memory usage, and disk space usage. Configure CloudWatch dashboards to provide a visual representation of the performance of each instance. This will allow you to easily monitor the performance of the system in real time. Splunk Monitoring is a platform for monitoring, searching, analyzing, and visualizing machine-generated data. You can use Splunk to monitor logs, metrics, and other data generated by the SAS Three-Tier Architecture. To monitor a SAS Three-Tier Architecture using , you can use the following steps: Splunk Splunk Install and configure the Splunk Universal Forwarder on each EC2 instance running the SAS Metadata Server, SAS Workspace Server, and SAS Stored Process Server. This will allow Splunk to collect log data from each instance. Configure Splunk to index the collected log data. Use Splunk to create dashboards and reports that provide insights into the performance of the system. These dashboards and reports can be customized to meet the needs of different stakeholders. Monitoring a SAS Three-Tier Architecture using CloudWatch, and Splunk is essential to ensure that the system is running optimally. By following the steps outlined in this article, you can monitor the performance of the different components of the system and gain insights that can be used to make informed decisions. Lead image . reference References: https://www.sas.com/en_us/home.html https://aws.amazon.com/ https://developer.hashicorp.com/terraform/intro https://www.splunk.com/

Powerful Data Analytics Software:  SAS Three-Tier Architecture Install on AWS Environment 

I have two kids and, if I'm being honest, I'm tired . all the time Of course, children are a gift, the miracle of life, blah blah blah. But they are also little energy vampires that demand your attention and affection at all times regardless of how feel and reflect all of your bad habits back at you like some backward version of the Mirror of Erised. you Hello, Parenthood Don't get me wrong, I love being a dad, but don't ever let anybody tell you that it isn't work. When you are responsible for bringing life into this world, you're also responsible for everything life is responsible for. It's not just about keeping them alive; you actually have to things like talk to them, spend with them, and teach them how to be . hard that do time Not-Bad Adults™ That shit is hard, and to do it right, you have to throw your whole self into it , and you know what also has a reputation of requiring your whole self ? every day every day Startup life. Starting Up I've worked almost exclusively for pre-Series A startups for my entire career (which basically means I'm perpetually busy and broke). But what startups lack in money and free time, they make up for in variety. You see, I've always considered myself to be a generalist. I get bored easily, so I to wear a lot of different hats and put my hands in a lot of different pots—and probably some other metaphors I can't think of. like Startups offer more opportunities to learn than you could possibly imagine, and when I was in my 20s that bargain was to maintain. Work never stopped, because learning never stopped. There was always something to do. New problems to solve, new technologies to learn, new experiments to run. easy And then I had kids. Finding Balance There is nothing quite so humbling as being completely responsible for the well-being of another person. Your time is no longer completely our own, and if you want to survive the , it's no longer your company's either. Parenthood Game I found pretty quickly that the startup life I had been living wasn't sustainable as a dad. Late nights and early mornings are hard to maintain between feedings, diaper changes, preschool drop-offs, and school bus schedules—never mind all that quality time kids (probably) need. So, for the first time in my life, I had to find balance—the "work/life" kind, not the "man on a wire" kind—and, let me tell you, it was freaking hard. Get Shit Done As every book about adulting tells us, bad habits are hard to break because they are so easy to engage in; and what's easier than simply on the computer or leaving the office? staying not When your time is wholly your own, it is easy to slack off at work. Coffee walks, long lunches, internet spelunking... there's time for it all because if you waste the day, you can always make up for it at home. incredibly Not as a parent. When you are working, you have to . If you want to maintain your sanity (and the precious few hours of sleep you are able to carve out), you don't have the luxury of doing the work later. Get it done because once you get home you need to take off the employee hat and doff the 'ole dad hat. get shit done now There are a thousand techniques to be more efficient at work (and just about as many books), but at the end of the day, you just need to take responsibility for your time. Learn to time block, find your flow state, say "no" to unnecessary distractions, and have a clear understanding of what your goals are for the day. You don't have to become a productivity robot—what's the point of work if you can't find any enjoyment in it?—but you should be mindful of how you use your time, and how your choices might affect you later on. Stay Healthy Like many dads, I struggle with this one. After all, the "dad bod" is a thing for a reason: once you have kids, it can be easy to let yourself go because you're juggling so many other balls. so But an unhealthy lifestyle is incompatible with both parenthood startup life. Startups can be both mentally and physically demanding, and kids have all of the energy in the world, so if you don't take care of yourself, you'll never be able to give what they need. and either Trust me, I'm not a health expert (just ask my scale), but if there's one thing I've learned it's that not taking care of my health has a negative effect on in my life. You can't be your best self if you feel like shit all the time; so do something about it. everything Get a standing desk. Take scheduled walks. Eat some green stuff. Meditate. There's no silver bullet, but the effects of making an effort are cumulative, and the more you make an effort, the better you will be able to manage it all. Set Boundaries The hardest lesson I've had to learn is that the work ends when the work day ends. When you've packed your bag and walked out of the building—or, in today's "I work from home so I'm always at work" world, when you shut off your computer—nothing short of a emergency should bring you back into work-brain until you log back in the next morning. true Kids learn boundaries not by what we say, but by what we do. The behaviors we reward and reject demonstrate what our limits are. true It's easy to that you value work-life balance, but when you check in code at 7 pm or send an email after midnight, it makes it clear that your words are just that: words. say I'm not saying that you can't work after "work hours." At a startup, that is impractical. What I am saying, though, is that you need to clearly define your limitations and then . Whether it's not responding to Slack messages after a certain time, or only dealing with email first thing in the morning, you must figure out what your boundaries are, and hold them as your life depends on them. live them Fire Your Phone This tip can be tough to swallow, but work for a startup, your phone. Know a good way to enforce those boundaries you set above? you not Delete all the work apps off your phone. Email? Bye. Slack? Gone. Jira? See ya. When I'm off the clock, I'm off the clock. Slack, Email, Jira, whatever other apps we've integrated into every facet of our lives... those things are for work, so I leave them there. By allowing work apps on your phone, you are allowing work into your personal life. We live in a world where we are often asked to leave our personal "problems" at the door when coming into the office, so it's only fair to ask the same thing in return. Worried about emergencies? In my experience, there are vanishingly few emergencies, but if there one, my team has my phone number and knows how to get ahold of me. real is the phone apps to the computer ones? , so dig out one of the old phones you have floating around a drawer and throw your work apps on . Give yourself the opportunity to lock work . If you can't do that, then you're never "off." Prefer I don't get it, but I recognize that everyone has their own preferences that away really Put Your Kids First I know it should go without saying, but you ; and if it feels like I'm flogging the whole "set boundaries" thing a bunch, it's because I am. Work is important, I'm not arguing against that. But my kids? have to put your kids first It's no competition. My oldest daughter gets on the bus at 8:12 in the morning, and she gets off it at 3:28 in the afternoon. My youngest needs to be dropped off at school by 8:30 and picked up by 3:30. Dance class is on Wednesday nights. Spaghetti Night is Monday, and Game Night is . Parent-teacher conferences, dance recitals, girl scout meetings, daddy-daughter dates, her as often as humanly possible Everyone has a life outside of work, and in mine my kids need my wife and I to show up for them I show up for work. I will make standup and retro, hit deadline, and when the shit hits the fan, I'll be the first one to sacrifice some sleep to keep the product on the rails. at least as well as every every But I'll also be there for my kids when need me—and they need me—because in the end (if I do things right), they will be in my life a hell of a lot longer than any startup possibly could. they how Just Keep Swimming It felt appropriate to conclude this with a quote from a Disney movie. Or is it Pixar? Is one of those Disney-Pixar things? I dunno. Doesn't matter... Finding Nemo *ahem* It's a lot easier to be Employee of the Month than Dad of the Year. Being a parent is ambiguous. There are no sprints, no backlogs, no retrospectives, and no 1:1s. You never know quite how you are doing, and even if you did, there's no way to it. All you can do is just keep swimming (or, if startup parlance is more your speed, keep chopping wood). measure My oldest daughter is turning 8 this year, and if I've learned anything in her near-decade of life it's that there are a of startups out there that I could join that will respect my boundaries, but only one . Finding balance is hard—and maintaining that balance even more so—but it's a hell of a lot easier when you know what you truly value. ton her Also published at . flower.codes

Raising Kids And A Startup

Criando hijos y una startup

Élever des enfants et une startup

बच्चों की परवरिश और एक स्टार्टअप

子育てとスタートアップ

Criar filhos e uma startup

Nuôi dạy trẻ em và khởi nghiệp

抚养孩子和创业

Application Monitoring: Closing Observability Gaps with Custom Metrics

is an open-source CLI tool that makes it easy to pipe data from one tool to another. At , we spent time determining if we could leverage Singer to programmatically send data from of their supported data sources (taps) to any of their supported data destinations (targets). Singer.io Airbyte any For the sake of this article, let’s say we are trying to build a tool that can do the following: Run any Singer tap or target Provide a UI for configuring and running those taps and targets Count the number of records synced in each run In the context of these goals, being able to use Singer programmatically means writing a program that can, for any integration: provide a UI with instructions on what information a user needs to input in order to configure that integration (e.g., host, password, etc). take those user-provided values and execute each integration. We know that the described requirements are not the use case that Singer sets out to solve, but nonetheless, we wanted to see if we could leverage Singer to bootstrap building out this case. Sure enough, we ran into some “gotchas” along the way. These gotchas illustrate some of the core primitives that a programmatic data integration tool requires. Integrations do not declare their configurations The Singer protocol does not . This means that, in order to use most Singer taps, you need to scour the entire implementation to figure out what properties it uses; depending on the complexity of the integration, this can be pretty painful. specify how an integration should define what inputs it requires Some integrations help out by specifying what the configuration should look like in a or in a . Even these lead to headaches. They often just list the fields that need to be passed in but do not explain what they mean, what their format is, or how to find them (good luck trying to find all the information you need to configure your Google Ads integration!). In other cases, they only list a subset, and then you have to discover the rest by reading the integration (e.g., doesn’t mention is_sandbox in the docs UPDATE: someone has now added this field in the readme with this ). readme sample config tap-salesforce PR These taps are great; we have happily used all of them, but because they do not specify what is required to configure them, they can’t be used programmatically. Specifically, our program needs to know that for the Postgres tap it requires the field’s hostname and port. Without this specification, the program cannot figure out how to build a valid configuration for an integration. This configuration is expensive to shim, because it requires engineering work for every single integration! No way to tell which Singer feature is compatible with which integration Singer has excellent around its core protocol. It also does a nice job defining the suite of special metadata that it supports. When you start actually using Singer, however, mapping these primitives onto your integrations is difficult. For example, replication-method sets whether all the data from the source should be replicated (full_table) or just the new or updated data (incremental). What is unclear is which taps actually support incremental or full_table or both. documentation Taps do not advertise, in a way that is programmatically consumable, which of these replication methods they support. Some of them mention it in their documentation, but ultimately that’s insufficient for the type of tool we want to build. So what happens when you request incremental from a source that only supports full_table? The behavior is undefined. Some taps will throw an error, some will just do a full refresh. Either way, from the point of view of the UI-based tool that we are trying to build, this isn’t really usable. The problem only gets hairier for some of the more niche metadata as well (e.g., view-key-properties). You either need to read the source or just try it out and see if the configuration works. This problem is adjacent to the configuration problem described in the previous section, and, similarly, requires a shim for every integration. Singer’s own secret menu If you’re from the West coast, you might be familiar with how In-N-Out Burger . While charming at a drive thru, secret menus can ruin your data integration. popularized the “secret” menu in fast food chains The Singer protocol has some of its own secret menu items. For example, we were parsing each message that a tap output into JSON using the declared schema in the Singer docs. We were trying to understand really well what messages were being sent between taps and targets, so we would fail loudly if anything was sent that did not match the documented message types. Then we started getting errors on ActivateVersionMessage. After spelunking in the source code for a bit, we found that this message type has existed in Singer as an experimental feature since 2017. A handful of the official Singer taps use it, but there’s no guidance on what you’re supposed to do with it (I suspect it is a feature used internally at Stitch–the paid, managed solution from the creators of Singer). If you’re building something programmatic on top of Singer, your choice is to just filter it out or let it pass and hope that stuff…just works, I guess? Handling this one case is not the end of the world, but it leaves you feeling uncertain what else is lurking in the protocol that might not play well with your system. Conclusion So to answer our original question, can we reasonably stretch the Singer to meet our product requirements? The answer is no. Doing so would require writing custom shims for every single Singer tap and target. Since the goal with data integrations is always to scale to more integrations, having to do any work on them per integration is very expensive. The Singer protocol is underspecified for this use case. This realization makes sense, because ultimately this is not the use case for which the protocol is trying to solve. Achieving these requirements depends on integrations declaring much more information about how they are configured and which features they support. We are tackling this problem at , so if you are looking for an OSS solution that makes it easy to move your data into a warehouse, instead of trying to roll your own on top of Singer, come check us out! Airbyte This article is meant to be the first in a pair of articles. The second will explore the engineering journey that we took to figure out where Singer should fit into our system. Previously published at https://airbyte.io/articles/data-engineering-thoughts/why-you-should-not-build-your-data-pipeline-on-top-of-singer/

Please Dont Build Your Data Pipeline using Singer

Eventually, you'll write a super handy script for interacting with Kubernetes – I have no doubt. Wouldn't it be stellar if you could that your script was officially part of the repertoire? pretend kubectl You can! is a , in the same fashion as . When you call it with a sub-command it doesn't intrinsically recognize, it tells you: kubectl multi-call binary git $ kubectl fu unknown command "fu" for "kubectl"

Did you mean this?
	run
	cp

Run 'kubectl --help' for usage. Error: Before it spits out the usage screen, however, does a bit of spelunking through each component directory in your , looking for an executable file named . We can use this to our advantage: kubectl $PATH kubectl-$COMMAND $ /Users/jhunt/bin:/usr/bin:/bin:/usr/sbin:/sbin

$ cat ~/bin/kubectl-fu $ kubectl fu
the kubernetes is strong with this one... echo $PATH #!/bin/sh echo "the kubernetes is strong with this one..." It doesn't matter what you write your program in; hack up some Bash, compile some Go or Rust, even write it in Perl or Python! As long as it is in , and has the executable bit set, you can pretend to be a core CLI author! $PATH Here's a more potent and useful example, which uses a bit of Go magic to handle the base64 encoding that Kubernetes puts on all of its Secrets. If you store secrets in Kubernetes, they get encrypted at rest, and are returned by the API encoded using the Base 64 scheme. While the , using the encoded data is a bit of a pain. particulars of the algorithm are interesting $ kubectl creds -o = aXQncyBhIHNlY3JldCB0byBldmVyeWJvZHku get secret jsonpath '{.data.password}' Luckily, we can use Go templates to format data we get from the API. What's more, the template language has the ability to decode base64-encoded data, natively! Here's a first attempt at extracting the data: $ kubectl secret creds -o template='{{.data.password | base64decode }}' 's a secret everybody. get it to That's mighty useful. If we package it up as the new sub-command, it gets 10x more useful! kubectl decode $ cat ~/bin/kubectl-decode kubectl get secret -o template= $ chmod 0755 ~/bin/kubectl-decode

$ kubectl decode creds
it #!/bin/sh exec '{{.data.password | base64decode }}' " " $@ 's a secret to everybody. Also seen here .

Stories, News & Mentions about SPLUNKINC

Splunkinc’s Stories on HackerNoon

Splunkinc's News & Mentions