Get the most recent info and news about SPLUNK INC on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Splunk Inc., together with its subsidiaries, provides software and cloud solutions that deliver and operationalize insights from the data generated by digital systems in the United States and internationally. The company offers Splunk Platform, a real-time data platform comprising collection, streaming, indexing, search, reporting, analysis, machine learning, alerting, monitoring, and data management capabilities. It also provides Splunk Solutions, such as Splunk Security solutions that enable cybersecurity teams streamline the security operations workflow, accelerate threat detection and response, enhance threat visibility, and scale resources to increase analyst productivity through machine learning and automation; Splunk IT Solutions that provide IT Operations teams visibility and control across cloud and on-premises environments; and Splunk Observability Solutions for building and maintaining infrastructure and applications. In addition, the company offers Ecosystem Solutions, which includes pre-built data inputs, workflows, searches, reports, alerts, custom dashboards, flexible UI components, custom data visualizations, and integration actions and methods, as well as Splunk On-Call, Splunk Infrastructure Monitoring, and Splunk SOAR solutions, which provides APIs, SDKs, and other interfaces that enable its ecosystem, including third-party developers, partners, and customers to build content that configures and extends Splunk solutions to accommodate specific use cases. Further, the company provides adoption and implementation services, education services, and maintenance and customer support services. It sells its offerings directly through field and inside sales, and indirectly through various routes to market with various partners. Splunk Inc. was incorporated in 2003 and is headquartered in San Francisco, California.

SPLUNK INC

splunkinc

Splunk announced layoffs impacting 7 percent of staff ahead of its deal to be acquired by Cisco for $28 billion.

Splunk Cuts 7% Of Staff; ‘Not A Result’ Of Cisco Deal, CEO Says

In the latest market close, Splunk (SPLK) reached $147.16, with a -0.03% movement compared to the previous day. The stock trailed the S&P 500, which registered a daily gain of 0.65%. On the other hand,

Splunk (SPLK) Stock Slides as Market Rises: Facts to Know Before You Trade

In the latest trading session, Splunk (SPLK) closed at $147.49, marking a -0.12% move from the previous day. The stock exceeded the S&P 500, which registered a loss of 1.43% for the day. Meanwhile, the Dow experienced a drop of 0.

Splunk (SPLK) Stock Moves -0.12%: What You Should Know

Goodson shares his thoughts on Atom Computing's new second-generation 1,225-qubit neutral atom quantum computer.

Atom Computing Announces Record-Breaking 1,225-Qubit Quantum Computer

It doesn't matter if you're a growth, value, income, or momentum-focused investor -- building a successful investment portfolio takes skill, research, and a little bit of luck. But what's the best way to find the right combination of stocks?

The Zacks Rank Explained: How to Find Strong Buy Computer and Technology Stocks

In this article, I will delve into whether artificial intelligence (AI), particularly the emergence of tools like generative AI (GenAI), will replace classical software outsourcing as we know it.

AI vs software outsourcing: An opportunity or a threat?

Cisco Systems' recent financial results were strong, but the planned acquisition of Splunk for $28 billion adds operational and financial risk without adding significant value. The acquisition ...

Cisco Stock: Outlook Amid $28B Splunk Deal (Rating Downgrade)

Drexel’s College of Computing & Informatics' Master of Science in Software Engineering (MSSE) is designed for students who hold a four-year bachelor's degree or master’s degree from a regionally accredited institution in computer science, software ...

Master's (MS) in Software Engineering

TRIPS is an important treaty under the WTO. The TRIPS Agreement represented a major expansion in the scope of copyright protection for computer software. TRIPS brought software under the purview of the Berne Convention by defining it as a literary work.

Preserving national integrity: Shunning software piracy

UiPath, Inc. is focused on building and managing automations and developing computer vision technology ... Splunk Real User Monitoring, and Splunk Synthetic Monitoring. It provides application programming interfaces, software development kits, and other ...

Which Is a Better Investment, UiPath Inc or Splunk Inc Stock?

What happens when you take Splunk, a wireless smart meter and an api interface? Splunking your entire home’s power main, without touching a single power wire! This allows a whole house view of all the power being drawn in real time. The only requirements are raspberry pi on a network, a splunk instance somewhere and a usb adapter plugged into the PI. Hold on, hold on, whats a smart meter? Recently the major power companies have been going around and replacing the old power meters on everybody’s house with a new fancy smart meter. This is essentially just a zigbee enabled power meter than allows power vans to roll down the street and collect all the readings quickly, without having a meter reader go house to house on foot. This page or the page for the a similar provider can be useful when identifying what kind of meter is installed. Then identify the usb dongle that is compatible to the smart meter, most sites have a list of those devices. Then order it up and get it registered via power company. A simple process however it requires paperwork and some lead time. The device that will be used here is the Rainforest EMU-2 Energy Monitoring Unit. / https://www.comed.com/SmartEnergy/SmartMeterSmartGrid/Pages/ForYourHome.aspx https://www.amazon.com/gp/product/B00BGDPRAI Enough of the prework time to dive in. The bulk of the work will be done with a python script that reads from the usb adapter, which is recognized as a serial port on the pi. Then we will install the python script as a service on the pi, which will allow it to run on boot and be easily restarted, and monitored. The python script: There are two requirements here pyRaven and Splunk_hec_handler, those can be found here: The splunk hec handler can also be found in pypi, however at the time of writing the version on pypi was broken and would not pip install, go after the source instead, then do the https://github.com/vavarachen/splunk_hec_handler https://github.com/nonspecialist/pyraven python setup.py install Great we have the requirements and the script. The script is pretty straight forward it connects to the serial port, it will listen to the serial firehose and parse when the data flows across. This is where the usb device is just terrible it has options to query the api but then there is a 4 second lag, during which time the same data will spit out automatically and then again from the query. It is really easier to sit there and listen to the firehose. Regardless the pyraven library does all the work and we are plugging in to it to grab the json object and pass it to a splunkhec function. The splunkhec function receives the data and appends the needed wrapper and posts the event. The posting of the event is then handled by the splunkhec library because why reinvent the wheel and deal with rest posts when we can use a library to do it. What this gives us is a nice json object being loaded into splunk. After we examine the data we can see that a simple splunk search, will give us the graph below. host=gate | timechart avg(event.raw_demand) as Load span=1m The service: In order for the pi to run this script automagically a service template is used and then installed. The service is assuming the the emu.py from above is going to be store in folder. /home/pi/ Alright the next step is to make sure the python script is executable and then install the service template and get it running. That can all be done with the snippet below. Thats all there is to it, the service is running the python script and outputting a whole bunch of nice data to play with. Once the data is acquired then the possibilities are endless for monitoring and reporting out of splunk for high usage or anomalous usage. ***BONUS: (JQ is required for parsing) In order to get the data in to splunk a HTTP collector token is needed. Use curl to generate a token: LINKS: _Splunk indexes and makes searchable data from any app, server or network device in real time including logs, config…_dev.splunk.com Use the CLI to administer HTTP Event Collector | Splunk _This repository allows you to access your Rainforest Automation Emu2 through it's serial interface …_github.com rainforestautomation/Emu-Serial-API _A Python Logging Handler for Splunk HTTP Event Collector (HEC). - vavarachen/splunk_hec_handler_github.com vavarachen/splunk_hec_handler _Interface with a Rainforest Automation RAVEn USB device - nonspecialist/pyraven_github.com nonspecialist/pyraven

Splunking Power Mains

Unlocking IaC Part 3: Your First Terraform Deployment!

IaC পার্ট 3 আনলক করা: আপনার প্রথম টেরাফর্ম স্থাপনা!

IaC Teil 3 freischalten: Ihre erste Terraform-Bereitstellung!

Desbloqueo de IaC Parte 3: ¡Su primera implementación de Terraform!

Déverrouillage d'IaC, partie 3 : votre premier déploiement Terraform !

IaC भाग 3 को अनलॉक करना: आपका पहला टेराफ़ॉर्म परिनियोजन!

IaC のロックを解除するパート 3: 初めての Terraform デプロイメント!

IaC 파트 3 잠금 해제: 첫 번째 Terraform 배포!

Desbloqueando IaC Parte 3: Sua primeira implantação do Terraform!

Открываем IaC, часть 3: ваше первое развертывание Terraform!

IaC Bölüm 3'ün Kilidini Açma: İlk Terraform Kurulumunuz!

Mở khóa IaC Phần 3: Triển khai Terraform đầu tiên của bạn!

解锁 IaC 第 3 部分：您的首次 Terraform 部署！

These are some AI-generated project names, courtesy of Namelix . Our naming story I write about developer innovation and new technologies for a VC firm called Project A. Every now and then, we develop projects that we’d like to open source. Coming up with names for these projects is always a challenge and each team has a different approach. I’d like to describe how we named a couple of our projects and what we learned along the way. One of these projects was initiated by our data engineering team and another by our back-end team. Despite having different use cases, both of these projects have a similar theme — they enable developers to efficiently process data from different sources and pipe it to different systems. Naming is hard but important Coming up with a name is never an easy decision. Babies, bands, top-secret projects, new products, and companies — someone, somewhere has to name them. Naming an open source project isn’t quite as “high-stakes” as naming a baby but it ain’t a walk in the park either. You can give an internal development project whatever crappy name you want, but if there’s a chance you’re going to go public with it, there are several reasons why you should put some effort into a decent name (besides simply avoiding ridicule). “A good name can help a company or product become successful, of course, but it can also help the lowliest code library find an audience, help formalize an informal process, and propel ideas about the world toward becoming talking points throughout it.” – Greg Leppert and Willem Van Lancker, Onym Developers can learn a lot from marketing pros Admittedly, we didn’t ask our marketing team for any tips about naming. They’re busy folks and we didn’t want to bother them with a highly technical project that we were going to share for free. For open source projects, you probably won’t need to worry about traditional marketing anyway — at least not when starting off. However, earlier in my career, I had collaborated with marketing teams on product naming and got to know some of the experts that they trusted. I started reading what these experts had written about naming and branding. Some of their ideas and concerns are still highly relevant for open source software. For example, I love this quote: “ “ We should lower our sights when it comes to naming. Because company names, like pretty much everything in life, fall on a bell curve. There are a few fantastic names, a whole bunch of okay-to-good names and a few truly awful names. The problem is, the surest way to end up with an awful name is by aiming for a fantastic one. — Doug Kessler, Velocity Partners It addresses how much time you can potentially waste trying to come up with the perfect name, and foreshadows some of the problems that we personally encountered. The first problem is… Who to involve in the naming process Whatever you do, you should keep the group small — I would recommend no more than 5 people. Any larger and you’re going end up with costly review cycles and discussions. Also, decide the group in advance and stick to it — gradually involving more and more people can undo any previous work and bring you back to square one. Or you could also leave the entire responsibility up to a senior member of your team. The creator of Maria DB, Max DB, and MySQL simply named those projects after his kids: “Maria”, “Max” and “My”. However, this strategy didn’t go so well for us. The backend project was first conceived a couple of years ago, by our CTO and a few of his colleagues. They were joking around and thought it would be fun to name the future project “Wayne” (no one had any children named Wayne). Now, I don’t want to offend any people with that name, but we felt it had certain connotations that we wanted to avoid. Not really the association we were going for — Wayne on YouTube (Image: YouTube) In all fairness to our CTO, “Wayne” was just a working title. He quickly needed a name so that he could discuss the thing that we wanted the team to build…without calling it “the thing”. However, most of us agreed that we needed a better name. Avoid a free-for-all naming frenzy Many development teams have flat hierarchies and try to involve everyone in big decisions. So, of course, we sought input from everyone in the team. Unfortunately, this strategy can lead to a lot of…shall we say…“noise”. Think of what happens whenever an organization decides to let the internet suggest a name for a product or project. As the New York Times wrote in 2016: happens. Nerds will be nerds. Boaty McBoatface And so it was, that names like “Crunchy McCrunchFace” and “BusinessObject McBusinessObjectFace” were some of the first suggestions when we tried to name our own projects (full disclosure, the second one was mine). People are also going to suggest names from popular culture such as “Gargamel”, “Tyrion”, and “Optimus Prime” — but good luck if you want people to find your “Tyrion” project in Google. Of course, it’s fun to do some free association but it probably wouldn’t do any harm to skip this part. Automating the brainstorming process Marketing teams have all sorts of complex methodologies that they like to use for this. For example, , Doug Kessler describes how they plotted words on scales such as “evocative vs descriptive”, “emotional vs rational” and “playful vs enterprisey”. They also arranged words that sent “the right signals” into a sphere of associations. in his article on rebranding If this stuff floats your boat, then by all means, give it a try — but I’m guessing you’d rather be coding than pinning names to an ideas board. And luckily, some of these processes have been automated by AI. While we were trying to figure out how to come up with a name, our CTO discovered an interesting tool called . According to their website, Namelix “uses artificial intelligence to create a short, brandable business name”. Namelix We weren’t after a business name but were game to give it a try. Namelix wants you to enter a few keywords about your business, so I entered the most generic words that related to our tool “backend” “database” “software”. I opted for a medium length name and chose a “name style”. There are 8 styles to choose from and I opted for “Misspellings” (like Lyft and Flickr). Some of the suggestions were baffling: “Scraink”, “Boonry”, or “Treesus”? If you need to come up with a name for your emo rap trio, the “Misspellings” option might work for you. But there was no way we were going to name our project “treesus”. I tried again with the “Compound Word” option (like “FedEx” and “Instagram”). Those suggestions were even odder: “Nannystate”, “Spyware”, or “Failapp”? In all fairness to Namelix, I did cherry-pick the funniest examples, but nonetheless, I couldn’t find anything that I liked and I didn’t want to spend hours tinkering with the settings. However, Namelix did help with clarifying the of words that we should consider. To see what I mean, have a look at their “name style” options: types Incidentally, our data engineering team had structured their naming process around on similar “naming styles”. Which leads me to the next point: Agree on your naming styles before you start brainstorming Our data engineering team had people propose names in categories such as “Italian words”, “Swahili words”, “Minerals”, “Characters from Roman Mythology”, “Plant Names” and “verbs”. If you agree on the categories that you want to use first, you can avoid the task of filtering through a whole bunch of random junk. Focus on Verbs The “verbs” category is particularly important — you want to build a list of verbs that describe what your project Even if none of them are suitable for a name, verbs are especially useful for triggering inspiration. does. To start off, our data engineering came up with simple verb-object pairs that described their project. Here are some examples: Import data Build dashboards Display data Transform data Extract, transform, load (ETL) data For each of those principal “actions”, they brainstormed further verb “associations”. For example, here are some examples of verbs that they associated with the “ETL” action. organize manage sort merge gather fold They then translated these verbs into Swahili. Swahili seems to be a favored language when it comes to naming projects. I suspect that Swahili words appeal to English and German speakers because they have a melodic texture and are pleasing to vocalize (maybe this has something to do with the ). bouba/kiki effect Again, here are some of the ETL verbs with their Swahili translations. organize: kuandaa extract: dondoo manage: kusimamia sort: aina merge: kuunganisha gather: kukusanya fold: mara The name that the data engineering team liked best was the last one: ‘mara’ — it was short, simple and easy to pronounce. Sometimes a good backstory provides the best option Our backend team had more of an intuitive approach. For them, it was more important that their name had a good backstory or anecdote that they could share. Our lead backend developer tried to think of metaphors for his team’s project. Then he hit upon a childhood memory — he remembered how much he loved making herbariums at school. The Herbarium — our lead developer’s childhood passion. He felt that the “herbarium” was a good metaphor for his team’s project. The project was an application for defining business objects independent of any database or tool. Different systems could then pull data whatever structure they require. For our lead backend developer, each business object was kind of like a pressed plant. Each plant has its own shape and attributes, but it gets flattened into a consistent form so that all of these plant samples can be stored and compared together in one place. He shared this idea with the team and everyone liked the metaphor. Some people felt like “herbarium” was a bit of a mouthful, so they shortened it to the more diminutive “Herbie”. But their naming quest wasn’t over yet… Do the Google Test Before you make a final decision on the name, make sure you find out what shows up on the search results page when you search for that word online. Branding professionals do this so that they can find out how stiff the competition is for the search results rankings. This isn’t a huge priority for most open source projects — you’re not trying to build a brand, you just want to share your project. However, you don’t want to find out too late that your project name has some unexpected connotations or associations (For example, Ikea had to deal with this problem when they ). expanded into Thailand Incidentally, our data engineering team was not immediately aware that “mara” also refers to an odd-looking mammal. That was until our Chief Data Officer went to the zoo with his family and spotted this sign: Fun fact: Here in Germany, there’s a very logical system for naming animals (as illustrated by this German animal names flowchart ) so in this neck of the woods, the “mara” is also called “Pampas Hare” (Pampashase). Fortunately, the Mara is kind of cute and it made for a great logo. Not everyone comes off so lucky, however. The point being, it’s better to be safe than sorry — even for a small open source project. What’s more important is searching on Github. Do the Github test Try checking the following URL patterns to see what comes up for your chosen name: https://github.com/<name> https://<name>.github.io We like to store our projects under their own organizations in Github so that they’re independent entities from the main “Project A” organization. These special projects consisted of several repos and we didn’t want them to get mixed up with all the other Project A repos. That’s why it was important for our data engineering team to get the account and for our backend team to get . https://github.com/mara https://github.com/herbie However, these short snappy names are often taken, but if you’re lucky, the name might belong to an inactive account. You can find out by reviewing the account activity. If there’s little to no activity, you might be able to claim the name according to . This was indeed the case for both the “mara” and “herbie” accounts. After contacting GitHub support, both names were made available for our projects. Github’s username policy Unfortunately for the backend team, they hadn’t paid enough attention to the general search results in Github. After everyone agreed on the name and set up the repos, people started discovering other “herbies” such as a German project and another from the University of Washington. Maybe “treesus” wasn’t such a bad idea after all? flat-file CMS small project While a double up is not ideal, it’s also not as critical as having a duplicate brand name. As long as the projects are in separate niches of the open source ecosystem, you can probably get away with it for a while. However, if another project with the same name becomes astronomically popular, your project risks drowning in a sea of content about its more well-known namesake. Check relevant top-level domain names This is not really a “must-have” but if you think that your project might gain traction, you could also check top-level domains that are popular with open source projects, such as “.org” or “.io”. Just be aware that some domains can come with a hefty price tag. At the time of writing, the domain “mara.io” was being sold for $1,999 — which definitely wasn’t an option for our data engineering team. Try out some usage examples As a final precaution, try to use the name in at least 5 sentences and make sure that there are no contexts where it sounds weird. For example, have a look at these sentences containing the term “Splunk” — the name of a popular application for analyzing machine-generated big data. “…to your new data, run the demo script…” “…you have the data but how will it be ?…” “…see our tutorial on Perforce Data…” “…check to see if knows about your script” “…both apps are enabled on this instance..” splunk splunked splunking Splunk Splunk Splunk is definitely a memorable name and a strong brand, but for British English speakers, there’s something disconcerting about using the word “Splunk” as a verb — it sounds somehow vulgar. A list of usage examples can help uncover such linguistic idiosyncrasies and it can also give you ideas on rules for correct usage. Further Resources Of course, I couldn’t finish a blog post about names without mentioning the indispensable naming resource “ ”. It’s a fantastic website which links to curated content about every aspect of the naming process, including many more tools that can help you with brainstorming. Onym I wish we had found out about this resource a lot earlier — we could have saved ourselves a lot of time!. But hopefully, these tips help you avoid some of the naming pitfalls that we encountered. Also, if you have any interesting stories about how you came up with a project name, let us know! A Disclosure Note On The Author and Project A Merlin blogs about developer innovation and new technologies at Project A — a venture capital investor focusing on early-stage startups. provides operational support to its portfolio companies, including developer expertise. As part of the IT team, he covers the highlights of Project A’s work helping startups to evolve into thriving success stories. Project A

How To Name Your Next Open Source Project

Ladies and gentlemen, I officially can say that the time has come. It has been an amazing 10 weeks of this crazy rollercoaster of a coding bootcamp. Full of emotions and enriches of friendships, I never expected this day to just be around the corner when I was watching another cohort experiencing the same adventure that I am about to undergo. If you are at Makers Academy, you’ve guessed it. My time has come for . Why is this moment so grand you ask for a student at Makers? Well, apart from the fact you are considered a senior… final project week Me trying to be as calm and confident as possible. You’ve finished all your foundation training of what the course has given you, and set out to practice. This is probably the last thing you’ll do at Makers This is the project, where in two weeks, you give it your best shot. This is where, when everyone heightens their senses and takes it up a notch because this is the moment to be proud off and prove it Although that sounds quite dramatic, I have to say it’s exciting and fun. It’s as exciting or entertaining as watching that scene in KungFu Panda where he’s completed all his training, and finally he is tested by his Shifu on using kung fu to be able to eat a bao. If you haven’t seen it, I recommend it. Before I start documenting this going to be unforgettable journey, . I think it’s the reason why I started blogging to begin with, to track all the lessons I have learn and continue to learn. I will say that I am writing this to document and remember my learning progress If that didn’t make sense, I guess what I mean is to l and . This is something I am practicing diligently because it stressed me out before when I didn’t and I realised I could be stopping the magic unknown moments that can come along the way. et the story unfold itself have no expectation on it Thus, I encourage you to do so. Am I excited about it? Yes. Am I going to give it my best shot? Yes. Am I ready for it? No, and who is? What is a final project (briefly): The final project is a two week schedule, where you are delegated into a team to produce a coding project base on your skills that you learn. Dependant on the team, you can choose what idea to work on and what languages to use. However, this project is different from the practice ones you’ve done because you have to consider things that you didn’t have to. Possible Things such as: The user flow for the project The goals of the project The way the project is going to be structured Once your final project is done, you have to present your project in front of coaches, peers, possible hiring managers, family and friends. The audience is a mix of support and a comedy crowd. Final projects finally give you the freedom of control to determine all of these based on what one learnt in the ten weeks. It’s a Makers Dream but it’s for sure easy to get your knickers in a twist. How the journey started Where does a cohort begin when it comes to dealing with final projects? A brainstorm session about ideas! On a Friday morning, the cohort gathered together to place either the topics, languages, product ideas or any other ideas on the board that a team could make. The ideas did not have to be great. It could be a prompt to something that didn’t have to be on the board. What this exercise did, for the coach (shoutout to Kay) and for us was to see what topics the cohort were thinking about it. Interesting stuff came up. Things like cryptocurrency, machine- , hacking a program that is meant to be unhackable, Tinder for animals. learning Yeah, Great stuff. Kay then grouped them into categories and based on a dot vote system, eliminated the categories that were not most of interest. The categories for our cohort were the following: Machine Learning Tricky Product Product Learning a new Language Technical Challenge From then onwards, it was base on our individual’s fate to choose our teams base on our interests. It definitely was a nerve-wrecking wait given that it all depended on one coach to determine our fate. However, she did not let me down. I don’t know whether it’s the fact she’s simply too good or have been having cameras everywhere of how this cohort has interacted, but she definitely placed me in a Dream team. The journey of TeamVEM: You’re probably wondering what is VEM, but that is not important. That will come later. Team VEM, including myself, is a team with the goal of creating a product. Whether it is a web app or mobile app, it has to be something somewhat solid. Lewis, Tom’s hand & Eva ❤ Our first group chat started late in the evening after a refreshed week of being alone doing tech tests. What we decided to do was to go through all the ideas that were categorised, and see which ideas we were not interested in. An elimination process was good to do first, given that it focused our group’s interest on the ideas. Once, we had a group of ideas, we then categorised them into themes of what we all were interested in. Our result was that we had one big concept following four topics that sparked our interests. Our overall concept was wanting to make something “interactive + educational” Our categories were Messaging/Social, Service, Tracker/Mapping and Software Service Although some topics interlinked or integrated with each other, it was good have bubbles of topics around so we knew the general interests of the group. We then met, Kay, the guider of our idea session, to specify our ideas further. It was a fulfilling experience to go through, as I felt I was experiencing what it was like to have an open yet professional discussion on what holes we saw in ideas and what potential we saw in others. However, we came to a conclusion that still needed a specification. “We want to build something interactive and educational with maps” Like the abstract form of the jelly shape surrounding all these ideas, this never started to wiggle in my mind, and it continued to wiggle throughout the rest of the evening. The fact that we weren’t solid on something yet, or knowing what product we were actually making spun the wheels in my mind. Since it had to make us think of our approach in this way: “How could I educate the user on a topic?” Which led to more questions such as: Why would you want to be educated on something? A long life lesson once reminded me that “Whys” are questions that go in circles if you can’t break down the question to other questions to hows or whats. Hence, I stopped immediately or needed the force or ping pong to stop my brain from spinning its gears. As it still wiggles my nerves, the positive part is that it can be still formed into something exciting or has the potential to still be nurtured. This reminded me of a script from the earlier mentioned movie, Kung Fu Panda that was the remedy to these thoughts. Master Oogway: “I cannot make it blossom when it suits me, nor make it bear fruit for me Shifu: But there are things you can control. I can control where the fruit will fall and where the seed is planted. That is no illusion! Master Oogway: No matter what you do with the seed, it will be a peach tree. You may wish for an apple tree, you may wish for an orange tree but it will always remain a peach tree. Shifu: But a peach tree cannot defeat Tai lung! Master Oogway: Maybe it can. If you are willing to guide it, to nurture it and believe in it. The lesson learnt here is to believe and nurture in the process and experience. To also nurture the idea, and believe in the idea no matter what flourishing thing it turns into. My move (research for the weekend): I believe it is an advantage if you come from a different background and enter in tech world. This is because you have more to share about the different places or experiences you come from. Hence, coming from a design background, I immediately knew that an inspiration for ideas of wanting to make “something educational, interactive with maps” could have the potential for data visualisation. Coming from a design background, looking at infographics designed around this topic would spark ideas of what innovative and creative topics people have chosen to do with maps, or in other ways. It also showed what potential data was out there for us to grab! Plus, it’s always a refreshing time for your eyes where you can look at printed material for immediate inspiration. Similar to design industry, the coding industry is a world that extracts from many different industries in the world because it’s a -based kind of industry. tool Conclusion of the start: Yeah, it’s great. It’s exciting. However, words of advice from having gone through the wave of emotions of the first day: If you are feeling nervous with the people you are working, don’t leap to conclusions about them just yet. They are growing just like you are, and they’ve probably learnt some major lessons along the same journey as you have. If you are feeling scared of being the weakest link, stop. It’s a team effort, and you will bring something more to the table which you don’t know yet. Don’t underestimate your ability or capability before the journey begins. If you are feeling a negative note on the whole situation, stop. Again, if you expect things already, you are blocking out the magical opportunistic things that could happen throughout the weeks. You mind is setting directions without looking at the whole map of where the project could take you. And if you are feeling all of these feelings, it’s okay to feel them. Like any big ‘milestone’ or ‘thing’, nerves are bound to come with something that is exciting. And with that, I sign off starting a new journey with #vems Just don’t let them linger on, for the sake of your sanity and your team’s vibe. Fun Fact of the Day: Other useful things to do when you start a final project, if you are stuck on what to do, is to talk about the following: What everyone’s goal is to get out from the project? What styles of standups and retros people are comfortable with doing. What people consider as important for the project? (Things like testing, wanting to set up the testing framework first, splunking to make sure everyone’s on the same page. What roles do people want to do? Yes, it may be awkward to put out there but By doing this, you are benefitting yourself and your team members to be more in sync with each other, before you even start. it’s necessary.

Episode 40: The final trials begin [Final Project Weeks]

Organizations now recognize the significant role of integrating security in the software development lifecycle. DevSecOps is a powerful approach that has emerged to achieve this goal. By incorporating security practices at the earliest stages of development, DevSecOps helps organizations identify and reduce vulnerabilities more effectively, minimize the risk of security breaches, and build more secure applications. This article will discuss the fundamental principles of DevSecOps and provide key steps for securing your organization’s CI/CD Pipeline. DevSecOps Principles : Organizational principles It is important to involve Security teams right from the beginning of a project to make sure that security requirements are given serious thought and attention early on. This step can prevent the discovery of security problems later in the development phase, which could be more expensive and time-consuming to fix. Involve Security Teams Early: In DevSecOps, it is important for Development, Security, and Operations teams to work together. By promoting a culture of shared responsibility for security, organizations can ensure that all team members play a part in creating secure applications and in identifying and solving security issues faster. Collaboration and Transparency: DevSecOps incorporates security practices early in the development process, also known as "shifting left." It helps organizations to detect vulnerabilities in the codebase before they escalate into expensive and time-consuming issues. Shift-Left: DevSecOps advocates for the inclusion of security procedures throughout the entire software development process, ensuring continuous security. This involves implementing automated security testing, constant monitoring, and regular security evaluations. Continuous Security: The concept of security chaos engineering involves deliberately introducing security-related problems and disruptions into a system to verify its ability to withstand them and reveal weaknesses. Security Chaos Engineering: By replicating actual security incidents, organizations can obtain significant insights into their system's performance during challenging conditions. This enables them to take proactive measures to address vulnerabilities and enhance their overall security position. Security Awareness/Training principles: To make security a top priority of your organization, provide your team members with security training and resources, encourage collaboration, and recognize that everyone is responsible for security. By doing this, you can cultivate a culture that prioritizes and integrates security throughout your organization. Foster a Security Culture: Developers should be encouraged to adopt secure coding practices like input validation, output encoding, and proper error handling. These practices will help mitigate security vulnerabilities in the code and reduce the risk of potential attacks. Adopt Secure Coding Practices: Gamification techniques can be adopted by organizations to encourage a security-focused culture and educate team members on security best practices. Gamification of Security Training: By transforming security training into an enjoyable and competitive experience, engagement and knowledge retention can be enhanced, ultimately promoting better security practices throughout the organization. Technical principles: Automating security processes can bring various benefits to organizations. These include increased efficiency, reduced human error, and the consistent application of security best practices. Automation: To maintain security during the development process, it is important to regularly run security checks and tests within the continuous integration and continuous deployment pipeline. This will ensure that security concerns are constantly evaluated and addressed. Integrate Security into the CI/CD Pipeline: Use tools like static application security testing (SAST) and dynamic application security testing (DAST) to check for weaknesses and potential security threats in your codebase. These automated security testing tools should be added to your organization's CI/CD pipeline to provide real-time security feedback. Implement Automated Security Testing: Securing the CI/CD Pipeline: Key Steps and Solutions To secure your CI/CD pipeline, protecting the Build environment is the first essential step. Only authorized personnel should access Build servers and infrastructure. To prevent unauthorized access, implement robust authentication methods, such as multi-factor authentication (MFA) with hardware keys. Additionally, keep the Build environment up to date with the latest security patches, and configure it according to the best industry practices. Secure the Build Environment: To safeguard your code repository, implement robust access controls that permit only the necessary access to authorized personnel, and monitor it frequently to identify any suspicious activities. Also, keep reviewing and updating user permissions and use encryption to protect sensitive data stored in the repository. Secure Code Repository: Hardcoded credentials, such as passwords or secret keys within the code, is a serious security risk as they can provide unauthorized access if the code is exposed. Hardcoded Credentials Scanner Solutions: Open-source solutions: TruffleHog - https://github.com/trufflesecurity/trufflehog - GitLeaks https://github.com/gitleaks/gitleaks - GitSecrets https://github.com/awslabs/git-secrets Enterprise solutions: GitGuardian - https://www.gitguardian.com/ SpectralOps - https://spectralops.io/ Bridgecrew - https://bridgecrew.io/secrets-scanning/ To automatically detect any potential security weaknesses in your code during the build process, integrate SAST tools into the CI pipeline. These tools can help you find problems such as insecure coding practices, input validation errors, and potential code injection vulnerabilities. Static Application Security Testing (SAST): Open-source solutions: SonarQube - https://www.sonarsource.com/products/sonarqube/ Semgrep CLI - https://github.com/returntocorp/semgrep Enterprise solutions: Veracode - https://www.veracode.com/products/binary-static-analysis-sast Checkmarx - https://checkmarx.com/cxsast-source-code-scanning/ - Coverity https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html To ensure the security of your application, you can use SCA tools to track third-party libraries and open-source components that you use. SCA tools can check for potential security risks such as outdated components, known vulnerabilities, and licensing issues. Software Composition Analysis (SCA): Open-source solutions: OWASP Dependency-Check - https://owasp.org/www-project-dependency-check/ - Nancy (GoLang) https://gonancy.dev/ - RetireJS (JavaScript) https://retirejs.github.io/retire.js/ - Safety (Python) https://github.com/pyupio/safety Enterprise solutions: - Black Duck Hub https://www.synopsys.com/software-integrity/security-testing/software-composition-analysis.html Mend - https://www.mend.io/sca/ Snyk - https://snyk.io/ In case your software uses containers, make sure to follow container security best practices, which include implementing minimal base images, scanning for vulnerabilities, and enforcing strict access controls. Additionally, it would be best if you used specialized container security tools, like container vulnerability scanners and runtime security monitoring solutions. Container Security: Open-source solutions: Trivy - https://github.com/aquasecurity/trivy - Grype https://github.com/anchore/grype - Clair https://github.com/quay/clair Enterprise solutions: Aqua Security - https://www.aquasec.com/products/software-supply-chain-security/ Sysdig Secure - https://sysdig.com/products/secure/ Snyk - https://snyk.io/product/container-vulnerability-management/ Incorporate security checks into your use of IAC tools like Terraform, CloudFormation, or Ansible. This is to ensure that the configuration of your infrastructure is indeed secure and free from potential security misconfiguration. Infrastructure as Code (IaC) Security: Open-source solutions: Checkov - https://github.com/bridgecrewio/checkov Tfsec - https://github.com/aquasecurity/tfsec KICS - https://kics.io/ Enterprise solutions: Bridgecrew - https://bridgecrew.io/infrastructure-as-code-security/ Prisma Cloud - https://www.paloaltonetworks.com/prisma/cloud/infrastructure-as-code-security - Snyk https://snyk.io/product/infrastructure-as-code-security/ To test your application in a real-time environment, add DAST tools to your CI/CD pipeline. These tools can uncover vulnerabilities that may only become visible when the application is executing, such as configuration issues, authentication problems, and access control concerns. Automated DAST tools should be used for thorough security testing. Dynamic Application Security Testing (DAST): Open-source solutions: OWASP Zed Attack Proxy (ZAP) - https://www.zaproxy.org/ - Nikto https://github.com/sullo/nikto - GoLismero https://github.com/golismero/golismero Enterprise solutions: Fortify WebInspect - https://www.microfocus.com/en-us/cyberres/application-security/webinspect Rapid7 InsightAppSec - https://www.rapid7.com/products/insightappsec/ - Checkmarx https://checkmarx.com/checkmarx-dast/ To enhance the security of your application and infrastructure, it is essential to continuously track and analyze their behavior. This can be achieved by implementing solutions for constant monitoring and logging. By doing so, you can quickly identify and respond to any potential security incidents that might occur. Continuous Monitoring and Logging: To facilitate this process, tools like SIEM (Security Information and Event Management) systems, intrusion detection systems, and log analyzers can be used to detect and alert you on security events. Open-source solution: ELK Stack (Elasticsearch, Logstash, and Kibana) - https://www.elastic.co/ Enterprise solutions: Splunk - https://www.splunk.com/en_us/products/enterprise-security.html Datadog - https://www.datadoghq.com/ To maintain the security and adherence of your CI/CD pipeline to industry standards and regulations, it is essential to conduct regular security audits and assessments. This includes examining the code, testing for potential vulnerabilities, and assessing the infrastructure. By implementing these measures, you can ensure that your pipeline is secure and compliant. Regular Security Audits and Assessments: Open-source solutions: OpenVAS - https://github.com/greenbone/ - BurpSuite Community https://portswigger.net/burp/communitydownload Enterprise solutions: Tenable Nessus - https://www.tenable.com/products/nessus QualysGuard - https://www.qualys.com/cloud-platform/ - BurpSuite Professional https://portswigger.net/burp/pro To be well-prepared for any security breach or incident, it is essential to create a clear incident response plan that lists the necessary steps to take. To ensure the safety of your application, your CI/CD pipeline must have the ability to roll back any deployments, patch any vulnerabilities, and immediately redeploy secure versions of your application. Incident Response Planning: Open-source solution: TheHive Project - https://thehive-project.org/ Enterprise solutions: IBM Resilient - https://www.ibm.com/products/qradar-soar Cortex XSOAR - https://www.paloaltonetworks.com/cortex/cortex-xsoar - D3Security SOAR https://d3security.com/enterprise-soar/ is a beneficial tool for DevSecOps teams. It's an open-source vulnerability management tool created by the OWASP community. Vulnerability Management and Reporting: DefectDojo This tool helps manage security vulnerabilities in the development process. One central space for all the information regarding defects is provided by DefectDojo. It makes it easier for teams to generate detailed reports and keep track of their progress over time. DefectDojo promotes clear communication about vulnerabilities, which ensures that security issues are resolved efficiently and promptly. As you can see, it is crucial to secure the CI/CD pipeline to implement a successful DevSecOps process. To achieve this, organizations need to incorporate security checks and tools throughout the pipeline. By doing so, they can identify vulnerabilities in advance and address them before they can cause security breaches. This helps to ensure that their applications are secure and comply with regulations. With the steps and practices outlined in this article, you can establish a robust and secure CI/CD pipeline for delivering software quickly and safely.

DevSecOps Principles and Key Steps for Securing the CI/CD Pipeline

SAS is an s a powerful analytics platform that allows users to perform complex data analysis and modeling tasks. To ensure high levels of performance and scalability, SAS offers a three-tier architecture that separates the different components of the system onto separate servers. This architecture allows for the metadata, data processing, and user interface layers to be installed on different servers for improved performance and scalability. In this article, we will discuss the steps to install a SAS three-tier architecture on an . AWS environment Provisioning EC2 Instances The first step in setting up a SAS three-tier architecture on AWS is to provision three EC2 instances. One instance will be used for the SAS Metadata Server, the second for the SAS Workspace Server, and the third for the SAS Stored Process Server. The size of the instances will depend on the size of your data and the expected load on your system. Here is an example Terraform code to provision three EC2 instances: provider "aws" { region = "us-east-1" } resource "aws_instance" "metadata_server" { ami           = "ami-0c94855ba95c71c99" instance_type = "t2.micro" tags = { Name = "metadata-server" } } resource "aws_instance" "workspace_server" { ami           = "ami-0c94855ba95c71c99" instance_type = "t2.micro" tags = { Name = "workspace-server" } } resource "aws_instance" "stored_process_server" { ami           = "ami-0c94855ba95c71c99" instance_type = "t2.micro" tags = { Name = "stored-process-server" } } Above code, we are using the AWS provider to set the region to us-east-1. We then define three aws_instance resources: metadata_server, workspace_server, and stored_process_server. Each instance has a specified Amazon Machine Image (AMI) and instance type and is tagged with a name for easy identification. Terraform Note that this is a basic example and you may need to modify the code to suit your specific requirements, such as adding additional configuration options, security settings, or networking components. Installing and Configuring the Required Software After provisioning the EC2 instances, the necessary software components need to be installed and configured on each of the instances. This includes installing the operating system, , database software, and SAS software. The specific versions of the software will depend on the version of SAS you are using and the requirements of your organization. Java Runtime Environment (JRE) Here is an example Ansible playbook for installing Java Runtime Environment (JRE), database software, and SAS software (SAS Preferred to installing GUI): name: Install JRE, Database Software, and SAS Software hosts: all become: true vars: java_version: "1.8.0" database_type: "oracle" database_version: "12c" sas_version: "9.4" tasks: name: Install JRE yum: name: "java-{{ java_version }}-openjdk" state: present name: Install {{ database_type }} Database Software yum: name: "{{ database_type }}{{ database_version }}" state: present name: Download SAS Software get_url: url: "https://example.com/sas/{{ sas_version }}/sas_{{ sas_version }}.tar.gz" dest: "/tmp/sas_{{ sas_version }}.tar.gz" name: Extract SAS Software unarchive: src: "/tmp/sas_{{ sas_version }}.tar.gz" dest: "/opt/sas" remote_src: true owner: sas group: sas name: Install SAS Software shell: "/opt/sas/setup.sh -deploy" args: chdir: "/opt/sas/install_data" Setting up the Metadata Server The SAS Metadata Server manages all metadata and user credentials for the SAS environment. Install and configure the SAS Metadata Server on the first EC2 instance. During the installation, you will need to specify the database and the location of the metadata repository. After the installation is complete, configure the metadata server using the SAS Management Console. Setting up the Workspace Server The SAS Workspace Server provides a connection point for SAS clients to execute SAS code and interact with data. Install and configure the SAS Workspace Server on the second EC2 instance. During the installation, you will need to specify the metadata server and the location of the workspace server data. After the installation is complete, configure the workspace server using the SAS Management Console. Setting up the Stored Process Server The SAS Stored Process Server provides a connection point for external applications to execute SAS programs and retrieve results. Install and configure the SAS Stored Process Server on the third EC2 instance. During the installation, you will need to specify the metadata server and the location of the stored process server data. After the installation is complete, configure the stored process server using the SAS Management Console. Configuring Network Settings After installing and configuring all the necessary components, network settings need to be configured to ensure that the three EC2 instances can communicate with each other. Firewall settings, security groups, and network interfaces need to be configured to allow communication between the three servers. It is important to follow best practices for securing your SAS environment on AWS. Testing the Installation Once all the components have been installed and configured, the SAS installation needs to be tested. Connect to the SAS Workspace Server using a SAS client application and run SAS programs on the Stored Process Server to verify that the results are returned correctly. By following the steps outlined in this article, you can install and configure a SAS three-tier architecture on AWS and achieve high levels of performance and scalability for your SAS environment. It is important to follow best practices for securing your SAS environment on AWS and to seek the assistance of a SAS expert if needed. Monitoring  a SAS Three-Tier Architecture using CloudWatch, and Splunk: CloudWatch Monitoring Amazon is a monitoring service for AWS resources and the applications running on them. You can use CloudWatch to collect and track metrics, collect and monitor log files, and set alarms. To monitor a SAS Three-Tier Architecture using , you can use the following steps: CloudWatch CloudWatch Enable CloudWatch logging for each EC2 instance running the SAS Metadata Server, SAS Workspace Server, and SAS Stored Process Server. This will allow CloudWatch to collect log data from each instance. Create CloudWatch alarms to monitor specific metrics for each instance. For example, you can create alarms to monitor CPU usage, memory usage, and disk space usage. Configure CloudWatch dashboards to provide a visual representation of the performance of each instance. This will allow you to easily monitor the performance of the system in real time. Splunk Monitoring is a platform for monitoring, searching, analyzing, and visualizing machine-generated data. You can use Splunk to monitor logs, metrics, and other data generated by the SAS Three-Tier Architecture. To monitor a SAS Three-Tier Architecture using , you can use the following steps: Splunk Splunk Install and configure the Splunk Universal Forwarder on each EC2 instance running the SAS Metadata Server, SAS Workspace Server, and SAS Stored Process Server. This will allow Splunk to collect log data from each instance. Configure Splunk to index the collected log data. Use Splunk to create dashboards and reports that provide insights into the performance of the system. These dashboards and reports can be customized to meet the needs of different stakeholders. Monitoring a SAS Three-Tier Architecture using CloudWatch, and Splunk is essential to ensure that the system is running optimally. By following the steps outlined in this article, you can monitor the performance of the different components of the system and gain insights that can be used to make informed decisions. Lead image . reference References: https://www.sas.com/en_us/home.html https://aws.amazon.com/ https://developer.hashicorp.com/terraform/intro https://www.splunk.com/

Powerful Data Analytics Software:  SAS Three-Tier Architecture Install on AWS Environment 

I have two kids and, if I'm being honest, I'm tired . all the time Of course, children are a gift, the miracle of life, blah blah blah. But they are also little energy vampires that demand your attention and affection at all times regardless of how feel and reflect all of your bad habits back at you like some backward version of the Mirror of Erised. you Hello, Parenthood Don't get me wrong, I love being a dad, but don't ever let anybody tell you that it isn't work. When you are responsible for bringing life into this world, you're also responsible for everything life is responsible for. It's not just about keeping them alive; you actually have to things like talk to them, spend with them, and teach them how to be . hard that do time Not-Bad Adults™ That shit is hard, and to do it right, you have to throw your whole self into it , and you know what also has a reputation of requiring your whole self ? every day every day Startup life. Starting Up I've worked almost exclusively for pre-Series A startups for my entire career (which basically means I'm perpetually busy and broke). But what startups lack in money and free time, they make up for in variety. You see, I've always considered myself to be a generalist. I get bored easily, so I to wear a lot of different hats and put my hands in a lot of different pots—and probably some other metaphors I can't think of. like Startups offer more opportunities to learn than you could possibly imagine, and when I was in my 20s that bargain was to maintain. Work never stopped, because learning never stopped. There was always something to do. New problems to solve, new technologies to learn, new experiments to run. easy And then I had kids. Finding Balance There is nothing quite so humbling as being completely responsible for the well-being of another person. Your time is no longer completely our own, and if you want to survive the , it's no longer your company's either. Parenthood Game I found pretty quickly that the startup life I had been living wasn't sustainable as a dad. Late nights and early mornings are hard to maintain between feedings, diaper changes, preschool drop-offs, and school bus schedules—never mind all that quality time kids (probably) need. So, for the first time in my life, I had to find balance—the "work/life" kind, not the "man on a wire" kind—and, let me tell you, it was freaking hard. Get Shit Done As every book about adulting tells us, bad habits are hard to break because they are so easy to engage in; and what's easier than simply on the computer or leaving the office? staying not When your time is wholly your own, it is easy to slack off at work. Coffee walks, long lunches, internet spelunking... there's time for it all because if you waste the day, you can always make up for it at home. incredibly Not as a parent. When you are working, you have to . If you want to maintain your sanity (and the precious few hours of sleep you are able to carve out), you don't have the luxury of doing the work later. Get it done because once you get home you need to take off the employee hat and doff the 'ole dad hat. get shit done now There are a thousand techniques to be more efficient at work (and just about as many books), but at the end of the day, you just need to take responsibility for your time. Learn to time block, find your flow state, say "no" to unnecessary distractions, and have a clear understanding of what your goals are for the day. You don't have to become a productivity robot—what's the point of work if you can't find any enjoyment in it?—but you should be mindful of how you use your time, and how your choices might affect you later on. Stay Healthy Like many dads, I struggle with this one. After all, the "dad bod" is a thing for a reason: once you have kids, it can be easy to let yourself go because you're juggling so many other balls. so But an unhealthy lifestyle is incompatible with both parenthood startup life. Startups can be both mentally and physically demanding, and kids have all of the energy in the world, so if you don't take care of yourself, you'll never be able to give what they need. and either Trust me, I'm not a health expert (just ask my scale), but if there's one thing I've learned it's that not taking care of my health has a negative effect on in my life. You can't be your best self if you feel like shit all the time; so do something about it. everything Get a standing desk. Take scheduled walks. Eat some green stuff. Meditate. There's no silver bullet, but the effects of making an effort are cumulative, and the more you make an effort, the better you will be able to manage it all. Set Boundaries The hardest lesson I've had to learn is that the work ends when the work day ends. When you've packed your bag and walked out of the building—or, in today's "I work from home so I'm always at work" world, when you shut off your computer—nothing short of a emergency should bring you back into work-brain until you log back in the next morning. true Kids learn boundaries not by what we say, but by what we do. The behaviors we reward and reject demonstrate what our limits are. true It's easy to that you value work-life balance, but when you check in code at 7 pm or send an email after midnight, it makes it clear that your words are just that: words. say I'm not saying that you can't work after "work hours." At a startup, that is impractical. What I am saying, though, is that you need to clearly define your limitations and then . Whether it's not responding to Slack messages after a certain time, or only dealing with email first thing in the morning, you must figure out what your boundaries are, and hold them as your life depends on them. live them Fire Your Phone This tip can be tough to swallow, but work for a startup, your phone. Know a good way to enforce those boundaries you set above? you not Delete all the work apps off your phone. Email? Bye. Slack? Gone. Jira? See ya. When I'm off the clock, I'm off the clock. Slack, Email, Jira, whatever other apps we've integrated into every facet of our lives... those things are for work, so I leave them there. By allowing work apps on your phone, you are allowing work into your personal life. We live in a world where we are often asked to leave our personal "problems" at the door when coming into the office, so it's only fair to ask the same thing in return. Worried about emergencies? In my experience, there are vanishingly few emergencies, but if there one, my team has my phone number and knows how to get ahold of me. real is the phone apps to the computer ones? , so dig out one of the old phones you have floating around a drawer and throw your work apps on . Give yourself the opportunity to lock work . If you can't do that, then you're never "off." Prefer I don't get it, but I recognize that everyone has their own preferences that away really Put Your Kids First I know it should go without saying, but you ; and if it feels like I'm flogging the whole "set boundaries" thing a bunch, it's because I am. Work is important, I'm not arguing against that. But my kids? have to put your kids first It's no competition. My oldest daughter gets on the bus at 8:12 in the morning, and she gets off it at 3:28 in the afternoon. My youngest needs to be dropped off at school by 8:30 and picked up by 3:30. Dance class is on Wednesday nights. Spaghetti Night is Monday, and Game Night is . Parent-teacher conferences, dance recitals, girl scout meetings, daddy-daughter dates, her as often as humanly possible Everyone has a life outside of work, and in mine my kids need my wife and I to show up for them I show up for work. I will make standup and retro, hit deadline, and when the shit hits the fan, I'll be the first one to sacrifice some sleep to keep the product on the rails. at least as well as every every But I'll also be there for my kids when need me—and they need me—because in the end (if I do things right), they will be in my life a hell of a lot longer than any startup possibly could. they how Just Keep Swimming It felt appropriate to conclude this with a quote from a Disney movie. Or is it Pixar? Is one of those Disney-Pixar things? I dunno. Doesn't matter... Finding Nemo *ahem* It's a lot easier to be Employee of the Month than Dad of the Year. Being a parent is ambiguous. There are no sprints, no backlogs, no retrospectives, and no 1:1s. You never know quite how you are doing, and even if you did, there's no way to it. All you can do is just keep swimming (or, if startup parlance is more your speed, keep chopping wood). measure My oldest daughter is turning 8 this year, and if I've learned anything in her near-decade of life it's that there are a of startups out there that I could join that will respect my boundaries, but only one . Finding balance is hard—and maintaining that balance even more so—but it's a hell of a lot easier when you know what you truly value. ton her Also published at . flower.codes

Raising Kids And A Startup

Criando hijos y una startup

Élever des enfants et une startup

बच्चों की परवरिश और एक स्टार्टअप

子育てとスタートアップ

Criar filhos e uma startup

Nuôi dạy trẻ em và khởi nghiệp

抚养孩子和创业

Which application metrics should you collect? I frequently engage with customers that are amid breaking their monolithic applications into smaller microservices. Many teams with also see this migration as an opportunity to make applications more observable. As a result, customers inquire which metrics they should monitor for a typical cloud native application. Previously, when a customer asked me how to instrument a service, I pointed them to the well known . But, I felt the response wasn’t thorough. A list of specific metrics to monitor can be helpful for teams building cloud native applications. This post is an attempt to provide a list of metrics to collect in a typical application. Not all the metrics listed below apply to every application type. For example, batch-like workloads rarely serve traffic, and resultantly, don't need to keep a log of requests-served. USE and RED methods The goal of this document is to help developers come up with the golden signals for their applications. Golden Signals, a term used first in the . Golden Signals are four metrics that will give you a very good idea of the real health and performance of your application as seen by the actors interacting with that service, whether they are final users or another service in your microservice application. Google SRE handbook Observability Cloud best practices recommend building systems that are observable. While the word observability (or “ ” as it is popularly known) doesn’t have an official definition, it is the measure of a system’s ability to expose its internal state. The three pillars of observability are logs, metrics, and traces. O11y Modern systems are designed to produce logs, metrics, and provide traces to help developers and operators understand its internal state. emit Push vs Pull Emitting metrics by exposing them on an externally accessible HTTP endpoint is gaining wider adoption thanks to developers adopting Prometheus for monitoring. In this model, Prometheus pulls metrics by scraping the application’s endpoint. /metrics When you run Node Exporter, it publishes metrics at http://localhost:9100/metrics Observability tools aggregate and analyze data from different sources to help you detect issues and identify bottlenecks. The goal is to use these system signals to improve its reliability and prevent downtime. AIOps products like can also detect anomalies using your application's logs, metrics, and traces (and other sources) and give you early signals to prevent a potential disruption. Amazon DevOps Guru Metrics to collect For an application to function as designed, the application and its underlying system have to be . Host metrics inform the operator of the host’s and infrastructure resource usage, like CPU, memory, I/O, etc. If you use Prometheus, collects this information automatically for you. healthy Node Exporter Host metrics rarely differ. Whether we run a process on an EC2 instance or a Raspberry Pi, we’re interested in the same metrics. Unlike host metrics, application metrics are unique to each microservice. Application metrics are supposed to provide the operator the information so they can do these things: Identify future areas of improvement by providing code-specific measurements. Application monitoring or APM tools provide measurements over a segment of time that developers can analyze. When the system fails, provide information for troubleshooting and prevention. In some cases, provide early signals to business. For example, if the application exposes, the it has processed in the last 60 minutes can be tracked using the monitoring system, rather than querying a relational database. orders There are several companies like application monitoring or APM companies like New Relic, DataDog that have products to aggregate application metrics using SDKs or agents. However, what they will not collect are the business specific metrics that only your application cares about. In order to create a list of relevant metrics for an application, its architects will need to determine a signal for its every key function. The hallmark of a microservice is that it does , therefore it shouldn’t have many key functions. Start by white-boarding the functions implemented in the code and creating a list of metrics that would help you gauge its performance (or its availability at the least). one thing well Most measurements you’ll do will fall under one of these categories: Counter As the name suggests, this value is incremented when a function runs. Example: total requests served Histogram A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. Gauge This type is metric tracks a value that increases or decreases over a period. Example: number of threads. With that background, let’s go through the list of common custom metrics developers use. Network activity These are the obvious metrics to track for any application that serves traffic. Network metrics tell you how much load is placed on the system. Over the time, these data points assist you when devising the scaling strategy for the system. Things you should include are: Request count by API type or page Requests total Transactions Concurrent, expired, and rejected sessions A watermark that records maximum concurrent sessions Average processing time A count by error type Resource usage It is a best practice to monitor a systems , which is a measure of your systems resource consumption. Every resource has a , beyond which additional stress causes performance degradation. Scalable and reliable systems are designed to never breach the breaking point. saturation breaking point However, simply collecting overall resource saturation at an application level is insufficient. You also need to look deeper at thread or resource pool level. Consider collecting these metrics: Number of processors, system CPU load, process CPU load, available memory, used memory, available swap, used swap, open file descriptor count. Total resources consumed by connection pools, thread pools, and any other resource pools. Total started thread count, current thread count, current busy threads, keep alive count, poller thread count, and connection count. Objects created, destroyed, and checked out, high-water mark, number of times checked out, Number of threads blocked waiting for a resource, number of times a thread has blocked waiting Common frameworks like Tomcat, Flask, etc. support exporting pre-defined metrics. For example, JMX already exposes a bunch of these metrics. See . AWS CloudWatch documentation Users Besides, serving the intended audience, bots or scripts flood internet facing web servers with requests. These automated requests can overload the system if unauthenticated requests are improperly handled (for example, not redirecting all unauthenticated requests to the authentication service and attempting to process an unauthenticated request). Here are user related metrics to collect: Authenticated and unauthenticated requests Demographics, authenticated and unauthenticated requests, usage patterns, Unsuccessful login attempts Some of these metrics may also come from your Load Balancer or ingress. Business transaction (for each type) If your application follows the microservices approach, then the code fulfills one function, at least that’s the idea. What are the key performance indicators for your app’s function? Define them and track these metrics. Should future releases cause performance regression, you’ll be able to detect it. Tracking these business metrics will help you track trends easily and avoid a cascading failure. Here are common things that services care about: Orders, messages, requests, transactions processed Success and failure rates. For a retailer, this could be the conversion rate. Service level agreements (like average transaction response time) If you still need help with identifying key metrics, ask yourself this question: In what ways can my application negatively affect the business even when it might appear to be healthy? Database connections Along with monitoring your database instances using database monitoring tools, consider collecting database connection health metrics in your application. This is especially helpful if your application uses a shared database. If your application encounters database connection errors but the database remains operational for other application, you know the problem is on the application side, and not the database. Consider recording these databases-related metrics: A count of thrown SQLException Number of (concurrent or maximum)queries Average query run time Data consumption Wherever you’re persisting data, you need to ensure that you’re going to go over your quotas and run out of space. Besides, monitoring on disk and in-memory data volumes, don’t forget to monitor the data your application stores in databases and caches. Cache health Speaking of cache, it is a best practice to monitor these metrics: Items in cache Get and set latency Hits and miss rates Items flushed Also, consider using an external cache such as Redis or Memcached. External services Keeping a track of how downstream services perform is also useful in understanding issues. Along with using timeouts, retries (preferably with ), and circuit breakers, consider monitoring these metrics for every external service your service's proper functioning depends on: exponential backoff Circuit breaker status Count of timeouts, requests Average response time or latency Responses by type Network errors, protocol errors Requests in flight A high watermark of concurrent requests. Granularity in metrics collection The frequency at which you publish and collect metrics depends on your business requirements. For a retailer, knowing traffic patterns by the hour and day is useful in scaling capacity. Similarly, a travel company’s traffic pattern are influenced by holiday schedules. Amazon EC2 provides instance metrics at 1-minute interval, which is a good start for critical metrics. Remember that there’s a cost attached to exposing, collecting, and analyzing metrics. Collecting unnecessary information in metrics can put a strain on the system and slow down troubleshooting. Consider giving the operator the control over the metrics your code should generate. This way, you can turn on specific metrics whenever needed. Conclusion Finding out which metrics to collect is an answer that only the most familiar with the code can answer. This post provides a list of metrics for you to get started. Are there any metrics that I have overlooked? Let me know at . @realz References https://giedrius.blog/2019/05/11/push-vs-pull-in-monitoring-systems/ https://www.splunk.com/en_us/blog/learn/sre-metrics-four-golden-signals-of-monitoring.html https://www.oreilly.com/library/view/learning-modern-linux/9781098108939/ https://www.oreilly.com/library/view/release-it/9781680500264/ Also published . here

Application Monitoring: Closing Observability Gaps with Custom Metrics

is an open-source CLI tool that makes it easy to pipe data from one tool to another. At , we spent time determining if we could leverage Singer to programmatically send data from of their supported data sources (taps) to any of their supported data destinations (targets). Singer.io Airbyte any For the sake of this article, let’s say we are trying to build a tool that can do the following: Run any Singer tap or target Provide a UI for configuring and running those taps and targets Count the number of records synced in each run In the context of these goals, being able to use Singer programmatically means writing a program that can, for any integration: provide a UI with instructions on what information a user needs to input in order to configure that integration (e.g., host, password, etc). take those user-provided values and execute each integration. We know that the described requirements are not the use case that Singer sets out to solve, but nonetheless, we wanted to see if we could leverage Singer to bootstrap building out this case. Sure enough, we ran into some “gotchas” along the way. These gotchas illustrate some of the core primitives that a programmatic data integration tool requires. Integrations do not declare their configurations The Singer protocol does not . This means that, in order to use most Singer taps, you need to scour the entire implementation to figure out what properties it uses; depending on the complexity of the integration, this can be pretty painful. specify how an integration should define what inputs it requires Some integrations help out by specifying what the configuration should look like in a or in a . Even these lead to headaches. They often just list the fields that need to be passed in but do not explain what they mean, what their format is, or how to find them (good luck trying to find all the information you need to configure your Google Ads integration!). In other cases, they only list a subset, and then you have to discover the rest by reading the integration (e.g., doesn’t mention is_sandbox in the docs UPDATE: someone has now added this field in the readme with this ). readme sample config tap-salesforce PR These taps are great; we have happily used all of them, but because they do not specify what is required to configure them, they can’t be used programmatically. Specifically, our program needs to know that for the Postgres tap it requires the field’s hostname and port. Without this specification, the program cannot figure out how to build a valid configuration for an integration. This configuration is expensive to shim, because it requires engineering work for every single integration! No way to tell which Singer feature is compatible with which integration Singer has excellent around its core protocol. It also does a nice job defining the suite of special metadata that it supports. When you start actually using Singer, however, mapping these primitives onto your integrations is difficult. For example, replication-method sets whether all the data from the source should be replicated (full_table) or just the new or updated data (incremental). What is unclear is which taps actually support incremental or full_table or both. documentation Taps do not advertise, in a way that is programmatically consumable, which of these replication methods they support. Some of them mention it in their documentation, but ultimately that’s insufficient for the type of tool we want to build. So what happens when you request incremental from a source that only supports full_table? The behavior is undefined. Some taps will throw an error, some will just do a full refresh. Either way, from the point of view of the UI-based tool that we are trying to build, this isn’t really usable. The problem only gets hairier for some of the more niche metadata as well (e.g., view-key-properties). You either need to read the source or just try it out and see if the configuration works. This problem is adjacent to the configuration problem described in the previous section, and, similarly, requires a shim for every integration. Singer’s own secret menu If you’re from the West coast, you might be familiar with how In-N-Out Burger . While charming at a drive thru, secret menus can ruin your data integration. popularized the “secret” menu in fast food chains The Singer protocol has some of its own secret menu items. For example, we were parsing each message that a tap output into JSON using the declared schema in the Singer docs. We were trying to understand really well what messages were being sent between taps and targets, so we would fail loudly if anything was sent that did not match the documented message types. Then we started getting errors on ActivateVersionMessage. After spelunking in the source code for a bit, we found that this message type has existed in Singer as an experimental feature since 2017. A handful of the official Singer taps use it, but there’s no guidance on what you’re supposed to do with it (I suspect it is a feature used internally at Stitch–the paid, managed solution from the creators of Singer). If you’re building something programmatic on top of Singer, your choice is to just filter it out or let it pass and hope that stuff…just works, I guess? Handling this one case is not the end of the world, but it leaves you feeling uncertain what else is lurking in the protocol that might not play well with your system. Conclusion So to answer our original question, can we reasonably stretch the Singer to meet our product requirements? The answer is no. Doing so would require writing custom shims for every single Singer tap and target. Since the goal with data integrations is always to scale to more integrations, having to do any work on them per integration is very expensive. The Singer protocol is underspecified for this use case. This realization makes sense, because ultimately this is not the use case for which the protocol is trying to solve. Achieving these requirements depends on integrations declaring much more information about how they are configured and which features they support. We are tackling this problem at , so if you are looking for an OSS solution that makes it easy to move your data into a warehouse, instead of trying to roll your own on top of Singer, come check us out! Airbyte This article is meant to be the first in a pair of articles. The second will explore the engineering journey that we took to figure out where Singer should fit into our system. Previously published at https://airbyte.io/articles/data-engineering-thoughts/why-you-should-not-build-your-data-pipeline-on-top-of-singer/

Please Dont Build Your Data Pipeline using Singer

Eventually, you'll write a super handy script for interacting with Kubernetes – I have no doubt. Wouldn't it be stellar if you could that your script was officially part of the repertoire? pretend kubectl You can! is a , in the same fashion as . When you call it with a sub-command it doesn't intrinsically recognize, it tells you: kubectl multi-call binary git $ kubectl fu unknown command "fu" for "kubectl"

Did you mean this?
	run
	cp

Run 'kubectl --help' for usage. Error: Before it spits out the usage screen, however, does a bit of spelunking through each component directory in your , looking for an executable file named . We can use this to our advantage: kubectl $PATH kubectl-$COMMAND $ /Users/jhunt/bin:/usr/bin:/bin:/usr/sbin:/sbin

$ cat ~/bin/kubectl-fu $ kubectl fu
the kubernetes is strong with this one... echo $PATH #!/bin/sh echo "the kubernetes is strong with this one..." It doesn't matter what you write your program in; hack up some Bash, compile some Go or Rust, even write it in Perl or Python! As long as it is in , and has the executable bit set, you can pretend to be a core CLI author! $PATH Here's a more potent and useful example, which uses a bit of Go magic to handle the base64 encoding that Kubernetes puts on all of its Secrets. If you store secrets in Kubernetes, they get encrypted at rest, and are returned by the API encoded using the Base 64 scheme. While the , using the encoded data is a bit of a pain. particulars of the algorithm are interesting $ kubectl creds -o = aXQncyBhIHNlY3JldCB0byBldmVyeWJvZHku get secret jsonpath '{.data.password}' Luckily, we can use Go templates to format data we get from the API. What's more, the template language has the ability to decode base64-encoded data, natively! Here's a first attempt at extracting the data: $ kubectl secret creds -o template='{{.data.password | base64decode }}' 's a secret everybody. get it to That's mighty useful. If we package it up as the new sub-command, it gets 10x more useful! kubectl decode $ cat ~/bin/kubectl-decode kubectl get secret -o template= $ chmod 0755 ~/bin/kubectl-decode

$ kubectl decode creds
it #!/bin/sh exec '{{.data.password | base64decode }}' " " $@ 's a secret to everybody. Also seen here .

Write Your Own Kubernetes Sub-Command [Part 9]

Well, well, exciting day… I have a real blog on it’s own unique domain name: . I paid the $0.01 entry month fee and am prepared to pay fullprice for EasyWP because it makes life easier not having to worry about permissions on wp-content/uploads or FTP access for WordPress to upload plugins or themes on Google Cloud Platform, yuck! Jare Cloud I’m going to use this blog to illustrate my days in a technical sense for a technical crowd. From interviews for places like or my spalunking for while they approach their capital event or even as they exist in open-source, before potentially becoming Intellectual Property of Coindex – such as my baby where on subscription fees, which was a more tame arbitrage bot – called, mysteriously, . Rebel Mouse Coindex Labs my many variations of crypto bots I might create fake volume for people updatedProveBot I’m negotiating fees for my first fake volume bot sub – a series of clients for a friend who’d approached me via my and seem to agree on $1k USD/month. Exciting! I’ll see if I can get them to pay me via Crypto Growth Hacker Telegram group, GitHub Sponsors, once I’m approved into their Beta. Who’s RebelMouse? I found them through a post, which is now >30 days old and removed – but mirrors this . The responsibilites match a lot of what I’ve done in the past – especially my time with OpenWater as a Technical Account Manager, where I’d supported as many as 20 client organizations at a time. I was responsible for 1-15+ client team members on each call, bringing them through a series of learning modules culminating in their successful use of the software. Being a Tech Evangelist for RebelMouse would be fun, productive and intellectually demanding – helping their client’s dev teams and executives to see the value in their software. I made it to the top 5-10 out of 1000s+ applicants on the short-list, and seemed to wow them to no end on the call – a CxO and a Director – and the next call with CEO and others will be scheduled for next week. Let’s Build! Sent them this email to improve my chances today: WeWorkRemotely.com LinkedIn job post – advertising for a Tech Evangelist role Who’s Coindex Labs? Ryan, CEO, approached me (unbeknownst to him) on my birthday with this glorious email: I’ve since come onboard as a founding team member – Lead Engineer responsible for Liquidity and Trading. Hodl! They’re approaching their capital event and might even bring me to cozy Atlanta in the process of bringing me on with a salary and other such benefits. They’ve been at their software for years – in as Lamen’s as I can, a (pantented!) neural evolutionary aritificial intelligence that trades smarter than any human can by reacting to market conditions in real-time based on a p2p novel search algorithm. Here’s the non-NDA teaser: Check out the final name on the Executive Bios, folks! So, we wait for Coindex to wow their funding interviews in early Oct and run into a tidy salary – while I simultanously wow some more executives with RebelMouse and run into a tidy salary, and max out the productivity on both positions while doing what I love for 60-80 hours a week – it sounds like a lot, but it’s sitting on my computer for 80 hours while my friends pull 100-120 on oil rigs for sustained periods. Comparatively, remote IT work is a breeze – I was in the Canadian Military Reserves as an Infrantryman for 2 years, and that’s objectively even worse than oil rigs (read: military not subject to labor laws). What I’m doing with the glorious proceeds? Get ’em to pay me via and put ’em on and watch my crypto compound interest and – for my immediate needs. Long story short, barring a Coindex exit sooner or BTC or ETH mooning, I’ll have enough ‘banked’ (yet unbanked) to retire comfortably in 1-2 years or in luxury in 3-4 years on DeFi stablecoin interest alone, not touching the capital. Hooah! Hodl! Build! Bitwage.com Compound.Finance borrow against the total – long ETH long BTC

Welcome to Jare.Cloud: Let's Make Some Introductions

Diy IoT is incredibly fun, and the barrier to entry is pretty low. There are wifi connected arduino boards out there to be had for $10–$20. Much of the time these boards will become packed with sensors and able to monitor their surroundings easily. Examples include temperature, humidity, barmoetric pressure, light and motion. Building further sensor gateways and gathering that data from the ‘dunios quickly shoots up the learning curve and gets developers quickly frustrated trying to gather from boards and process the data. Using a couple libraries one can quickly and simply get this data pushed into a personal splunk instance. This is done through a simple web post to the Splunk Http Event Collector or “hec.” From there the sky is the limit. https://www.adafruit.com/category/35 First things first what is an arduino? Arduinos are an entire clarification of microcontoller boards. Some are configured bare bones and others are esp8266 chips with extra memory space and breakouts. The boards used in this explanation are Adafruit “Huzzah” 8266 breakouts. These are used because of cost of materials, an arduino Uno board is the entry level board for $20. It is a bare microcontroller which would then require a second “shield” in order to bundle a wifi controller onto it. It is much easier to get a simple cheap huzzah. Adafruit has a great community of tutorials and getting started guides to get the Arduino IDE downloaded and configured, and then installing the proper libraries to support the huzzah board. Those libraries are then leveraged for the http and wifi clients. Take a look, get the IDE configured and the boards added here. https://www.arduino.cc/ https://www.adafruit.com/product/2471 https://learn.adafruit.com/add-boards-arduino-v164?view=all Before diving into the Arduino “sketch code”, it should be understood the kind of code we are going to see. Arduinos are pretty much a friendlier “C.” Much of the strongly typed language is there, however String is available and the entire compiling and upload process is done through a nice gui. It can also be done through makefiles and the like but the debugger is at least a little bit better in the Arduino IDE. It’s recommended to start there. A result of the C roots is that working with http requests is going to be a little less than user friendly, but not terrible. The http headers will need to be added and then the content length summed up and finally the entire message bundled up before submitting the http post request. While this isn't the end of the world if someone is not familiar with how http works it could be confusing. Don’t worry, take some time read the code, it will all make sense. Just over 100 lines of code, hopefully that was’t too bad. The majority of the work is broken out into a couple functions. The Loop() function is constantly running because that is how arduino works. Setup() is run once on boot. Then everything else is only run when called. The Loop() gathers up some dummy data and passes it the splunkpost() . If there was a more elaborate data gather needed it could be easily called from the loop and then returned to be sent on to splunkpost(). Splunk post receives all the parameters that were configured at the beginning for the Splunk server, HEC, and some metadata. There is a gotcha in that the http collector is hard coded in the firmware, there are more elegant and configurable ways to do this, however for this purpose it is fine. it should be noted that a rebuild and reflash would be needed if the HEC were to change. Thats all folks! An arduino connected to wifi network and then sending that data directly into Splunk via HEC. All without the need for anything else, no data gateways no serial bit banging between shields to push the sensors to the wifi board. Just sensor data from a DiY arduino going into the robust awesomeness that is Splunk. From Splunk all the analytics number crunching and alerting can be done. Hopefully this was descriptive enough to the wheels turning and the data flowing. Links: _Open-source electronic prototyping platform enabling users to create interactive electronic objects._www.arduino.cc Arduino - Home _Adafruit Industries, Unique & fun DIY electronics and kits Adafruit HUZZAH ESP8266 Breakout ID: 2471 - Add Internet…_www.adafruit.com Adafruit HUZZAH ESP8266 Breakout _Version 1.6.4 of the Arduino IDE introduced official support for adding third party boards (like the Adafruit Flora and…_learn.adafruit.com Overview | Adding Third Party Boards to the Arduino v1.6.4+ IDE | Adafruit Learning System _The HUZZAH ESP8266 breakout is what we designed to make working with this chip super easy and a lot of fun. We took a…_learn.adafruit.com Overview | Adafruit HUZZAH ESP8266 breakout | Adafruit Learning System _Adafruit Industries, Unique & fun DIY electronics and kits : Sensors - Tools Gift Certificates Arduino Cables…_www.adafruit.com Sensors : Adafruit Industries, Unique & fun DIY electronics and kits _The token admin can be the Splunk Enterprise or self-service Splunk Cloud admin, or a different person who does not…_docs.splunk.com About the Splunk HTTP Event Collector - Splunk Documentation

Arduino, Meet Splunk

Let’s face it, we have have been writing software for decades now. We can no longer claim to be a new domain and pass-off nascency as an excuse for shipping sloppy code. Most of us can tell the difference between good code and bad, but how do we go about creating beautiful instead of merely reading about it? software Well, let’s make an attempt here. This article is not going to teach you the zen of software maintenance. Rather it is just a dump from memory, sorted in order to ensure data persistence. Hopefully, some value is realized along the way. pseudo-random Empathize through code Speak so others will listen … write so others will read There is elegance in brevity. Code golfing might help you win , but the will always be missing. The empty chasms serve as bigger obstacles to collaboration than the longest of obfuscation blocks. JS1k lines-not-written Even when writing a simple wrapper for data retrieval, I have always found it useful to leave comments as markers along the trail. A few well-placed comments can save numerous man-hours of backtracking and spelunking, especially in highly collaborative teams. When the number of developers involved, , exceeds one, the need for clarity becomes non-negotiable for large projects. The catch is that is not an instantaneous measurement. Even if it is the same person reviewing code a few months later, becomes greater than 1 and the rule still applies. n n n (Don’t) Rewrite to update Projects which neglect the need to be inclusive rarely get updates. Rather, functionality updates and even minor fixes are delivered via system rewrites. Like many programmers, I sometimes jump to write fresh code rather than read old ones. This may be understandable as a side-effect of not following the previous section, but it is still a sub-par decision when developing software. Old code probably comes with many guarantees that your new solution does not. For one, the existing code works, while yours is still in the domain. We have version control software for a reason, use them. theoretically-faster Delete your way to perfection Every block of stone has a statue … the task of the sculptor is to discover it.– Michelangelo, probably Be proud of removing lines of code. I look forward to days when my commits show more red than green. Verbosity is a not a required dependency for clarity. In fact, clearing out decision paths and trimming circuitous execution cycles improve readability and reduce the search space as future bugs emerge. git no-longer-relevant If your manager ever asks you how many lines of code you have added to a project, I have only one piece of advice — run; . fast and in the opposite direction Abstractions are not forever We all love abstractions, they save us from repetition, allow us to stand on the shoulders of giants and are the secret to becoming a 10x rockstar-ninja. Okay, jokes aside, I have to admit, abstractions are really valuable tools. They allow us to leverage existing code as assumptions that can be when writing higher-level code. abstracted away However, you have probably also heard about . If not, the link is right there, I’ll wait, Joel is a much better writer than me. The Law of Leaky Abstractions Understanding the costs of abstraction is a crucial step in the journey of a developer. Learning to look for hidden costs is a crucial step in growing as a programmer. The earlier the realization is made, the easier the wakening from deceptive fantasies. One of my answers to is the opinion that the raw accumulation of practice becomes indistinguishable from innate talent. Peter Thiel’s contrarian question directed By making a concerted effort to write better code with each passing day, my hope is to get to a point where developing software has a much tighter execution step from thought to silicon. , I work at and usually blog about in building data-powered AI for ecommerce companies. A note before you leave Semantics3 my technical adventures To know more, check out , or to stay updated. our blog follow me on Medium

Towards better code , Thoughts on improving

ree ne wo con. TH O T Chicago's Hacking Conferencewww.thotcon.org THOTCON - Chicago's Hacking Conference There is a “super secret” hacker conference that has been held for the last 8 years in chicago. This is an information security conference akin to Defcon, Shmocon, and Derbycon. These conferences, are basically a whole bunch of information security nerds gathering together to talk about their trade, share some tips and tricks, swap stories, drink beer and generally not cause too much of a ruckus. However, the news would have you believe otherwise with a clickbait headline like: _Nick Percoco ushers us inside, and we are issued blinking electronic badges. The badges are basically mini-computers…_www.nbcchicago.com Thousands Attend Secret Hacking Conference in Chicago What is Thotcon then? It is a grassroots conference much like any other professional conference. There are a few major differences that sets Thotcon apart from everyone else. None of the talks are recorded. This allows for sensitive topics to be discussed without lawyers being worried, about proper disclosures and whatnot. There are no corporate sponsorships for the conference itself, this helps keep corporate bias out of the conference. this helps keep it grassroots and grassroots is part of the draw. The afterparty is sponsored, but that is for entertainment and cost purposes. Location is only disclosed to attendees and speakers but its in Chicago. This adds to the super secret hacker persona, but it is also just fun. I have attended this conference for 5 years and this year was the 8th year that the conference has existed. I have been a speaker at a worldwide conference with 3000 attendees (shameless splunk conf plug) The in house mini confs, mini hacks and full fledged hackathons for my company. c7five is the main OPER(ater) of the conference, he puts his heart into the conference and believes in it. It is refreshing to see someone that has been doing this for 8 years still have the passion driving the conference. I have been on many sides of the conference/meetup/event curtain. I have given a talk at a worldwide multi thousand person conference I can relate to some of the difficulties that a large event can be, there are months of planning that are involved. The logistics are scary complicated, and you want all of the talks to be amazing. Even when I put on a small meetup with only 20–30 people in the confines of my office the work was more than I could anticipate. https://conf.splunk.com/speakers/2014.html#search=fettis& The main problem is with the tech/security sector, the bleeding edge is all on twitter or reddit or irc. When a call for papers happens, it has to be months before the conference actually happens. By that time the bleeding edge is already forgotten about. This presents a difficult decision to try and host a technology conference that still has all of the drive and enticement to get people in the doors. What are the talks about then? Luckily Thotcon retains its hacker and devious roots, hackers love to talk and to brag. One of the benefits about the eyes only aspect of the conference is that government type or contractors can come in and talk about the really cool things they have done that they are not allowed to talk about, not even on twitter. Then, there are philosophical, ethical and theoretical type topics that spark debates, even more so once a couple beers are consumed and the real fun stories begin to flow. Also the conference attracts a wide range of attendees, some still in college and some with 20+ years of experience. If we are talking about a technological field this is about as diverse as it gets. Having a diverse audience can make networking a little difficult. This year a couple of times during the conference announcements were made instructing folks hiring to announce themselves and then people looking for a job to go flock to them. This was a novel idea and I think one that needs to be furthered again at the conference and at other conferences. Sometimes people are in fact attending to find a job, while others are just looking to hang out with friends. Then the people like me who attend and then only talk to a couple people, well because I’m shy like that. It can be difficult for people in my situation to get out of their shell, and if I had a good solution to that I would share it. Finally there is a final networking event known as the after party, where everyone comes back to the conference space, there is beer food and music, and everyone has a grand time. I am writing this post while that party is happening so I bet you can guess who did not attend and get their drink on? Every conference has a badge. The Thotcon badge for the last 4 years have been a microcontroller built in to the badge with a purpose or a game. A standard microcontroller is used in the badge that allows rewrite and reuse of badge, this also allows for fun badge games and exploration of the badge’s code through all the means available to a group whose skillset is exactly that. There is also another puzzle in the program with hints and clues hidden throughout it. The first person to solve it each year receives a gold badge that entitles them free admission to Thotcon for life. badge 7 is missing (its in my desk elsewhere) In order to attend a conference you need a badge, the badges are assembled locally at workshop88 which is a hackerspace/makerspace in the western suburbs of chicago While, they are designed and assembled locally Thotcon is also contributing to supporting local business by doing all of the badge work through workshop88. This is another facet of the hacker community that goes unnoticed to anyone on the outside. Hackers really do care about each other and there truly is that community around them. http://workshop88.com/ Finally the conference is one of the more unique and fun conferences that I attend yearly. On day 1 (4-may-2017) I was greated at the registration booth by a woman dressed in star wars leggings saying “May the fourth be with you”. Then, day 2 kicks off with mariachi band playing in honor of cinco de mayo, and the tequila is already flowing. why? because its cinco de mayo, thats why. I have been calling myself a hacker for over 15 years and I am never more proud to be on than when I get to attend such an event with other kin.

THOTCON 0x8

Git in 2016

Little Brother by Cory Doctorow, is part of the HackerNoon Books Series. You can jump to any chapter in this book . THE COPYRIGHT THING here THE COPYRIGHT THING The Creative Commons license at the top of this file probably tipped you off to the fact that I've got some pretty unorthodox views about copyright. Here's what I think of it, in a nutshell: a little goes a long way, and more than that is too much. I like the fact that copyright lets me sell rights to my publishers and film studios and so on. It's nice that they can't just take my stuff without permission and get rich on it without cutting me in for a piece of the action. I'm in a pretty good position when it comes to negotiating with these companies: I've got a great agent and a decade's experience with copyright law and licensing (including a stint as a delegate at WIPO, the UN agency that makes the world's copyright treaties). What's more, there's just not that many of these negotiations -- even if I sell fifty or a hundred different editions of Little Brother (which would put it in top millionth of a percentile for fiction), that's still only a hundred negotiations, which I could just about manage. I the fact that fans who want to do what readers have always done are expected to play in the same system as all these hotshot agents and lawyers. It's just to say that an elementary school classroom should have to talk to a lawyer at a giant global publisher before they put on a play based on one of my books. It's ridiculous to say that people who want to "loan" their electronic copy of my book to a friend need to get a to do so. Loaning books has been around longer than any publisher on Earth, and it's a fine thing. hate stupid license I recently saw Neil Gaiman give a talk at which someone asked him how he felt about piracy of his books. He said, "Hands up in the audience if you discovered your favorite writer for free -- because someone loaned you a copy, or because someone gave it to you? Now, hands up if you found your favorite writer by walking into a store and plunking down cash." Overwhelmingly, the audience said that they'd discovered their favorite writers for free, on a loan or as a gift. When it comes to my favorite writers, there's no boundaries: I'll buy every book they publish, just to own it (sometimes I buy two or three, to give away to friends who read those books). I pay to see them live. I buy t-shirts with their book-covers on them. I'm a customer for life. must Neil went on to say that he was part of the tribe of readers, the tiny minority of people in the world who read for pleasure, buying books because they love them. One thing he knows about everyone who downloads his books on the Internet without permission is that they're , they're people who love books. readers People who study the habits of music-buyers have discovered something curious: the biggest pirates are also the biggest spenders. If you pirate music all night long, chances are you're one of the few people left who also goes to the record store (remember those?) during the day. You probably go to concerts on the weekend, and you probably check music out of the library too. If you're a member of the red-hot music-fan tribe, you do lots of that has to do with music, from singing in the shower to paying for black-market vinyl bootlegs of rare Eastern European covers of your favorite death-metal band. everything Same with books. I've worked in new bookstores, used bookstores and libraries. I've hung out in pirate ebook ("bookwarez") places online. I'm a stone used bookstore junkie, and I go to book fairs for fun. And you know what? It's the same people at all those places: book fans who do lots of everything that has to do with books. I buy weird, fugly pirate editions of my favorite books in China because they're weird and fugly and look great next to the eight or nine other editions that I paid full-freight for of the same books. I check books out of the library, google them when I need a quote, carry dozens around on my phone and hundreds on my laptop, and have (at this writing) more than 10,000 of them in storage lockers in London, Los Angeles and Toronto. If I could loan out my physical books without giving up possession of them, I . The fact that I can do so with digital files is not a bug, it's a feature, and a damned fine one. It's embarrassing to see all these writers and musicians and artists bemoaning the fact that art just got this wicked new feature: the ability to be shared without losing access to it in the first place. It's like watching restaurant owners crying down their shirts about the new free lunch machine that's feeding the world's starving people because it'll force them to reconsider their business-models. Yes, that's gonna be tricky, but let's not lose sight of the main attraction: free lunches! would Universal access to human knowledge is in our grasp, for the first time in the history of the world. This is not a bad thing. In case that's not enough for you, here's my pitch on why giving away ebooks makes sense at this time and place: Giving away ebooks gives me artistic, moral and commercial satisfaction. The commercial question is the one that comes up most often: how can you give away free ebooks and still make money? For me -- for pretty much every writer -- the big problem isn't piracy, it's obscurity (thanks to Tim O'Reilly for this great aphorism). Of all the people who failed to buy this book today, the majority did so because they never heard of it, not because someone gave them a free copy. Mega-hit best-sellers in science fiction sell half a million copies -- in a world where 175,000 attend the San Diego Comic Con alone, you've got to figure that most of the people who "like science fiction" (and related geeky stuff like comics, games, Linux, and so on) just don't really buy books. I'm more interested in getting more of that wider audience into the tent than making sure that everyone who's in the tent bought a ticket to be there. Ebooks are verbs, not nouns. You copy them, it's in their nature. And many of those copies have a destination, a person they're intended for, a hand-wrought transfer from one person to another, embodying a personal recommendation between two people who trust each other enough to share bits. That's the kind of thing that authors (should) dream of, the proverbial sealing of the deal. By making my books available for free pass-along, I make it easy for people who love them to help other people love them. What's more, I don't see ebooks as substitute for paper books for most people. It's not that the screens aren't good enough, either: if you're anything like me, you already spend every hour you can get in front of the screen, reading text. But the more computer-literate you are, the less likely you are to be reading long-form works on those screens -- that's because computer-literate people do more things with their computers. We run IM and email and we use the browser in a million diverse ways. We have games running in the background, and endless opportunities to tinker with our music libraries. The more you do with your computer, the more likely it is that you'll be interrupted after five to seven minutes to do something else. That makes the computer extremely poorly suited to reading long-form works off of, unless you have the iron self-discipline of a monk. The good news (for writers) is that this means that ebooks on computers are more likely to be an enticement to buy the printed book (which is, after all, cheap, easily had, and easy to use) than a substitute for it. You can probably read just enough of the book off the screen to realize you want to be reading it on paper. So ebooks sell print books. Every writer I've heard of who's tried giving away ebooks to promote paper books has come back to do it again. That's the commercial case for doing free ebooks. Now, onto the artistic case. It's the twenty-first century. Copying stuff is never, ever going to get any harder than it is today (or if it does, it'll be because civilization has collapsed, at which point we'll have other problems). Hard drives aren't going to get bulkier, more expensive, or less capacious. Networks won't get slower or harder to access. If you're not making art with the intention of having it copied, you're not really making art for the twenty-first century. There's something charming about making work you don't want to be copied, in the same way that it's nice to go to a Pioneer Village and see the olde-timey blacksmith shoeing a horse at his traditional forge. But it's hardly, you know, . I'm a science fiction writer. It's my job to write about the future (on a good day) or at least the present. Art that's not supposed to be copied is from the past. contemporary Finally, let's look at the moral case. Copying stuff is natural. It's how we learn (copying our parents and the people around us). My first story, written when I was six, was an excited re-telling of Star Wars, which I'd just seen in the theater. Now that the Internet -- the world's most efficient copying machine -- is pretty much everywhere, our copying instinct is just going to play out more and more. There's no way I can stop my readers, and if I tried, I'd be a hypocrite: when I was 17, I was making mix-tapes, photocopying stories, and generally copying in every way I could imagine. If the Internet had been around then, I'd have been using it to copy as much as I possibly could. There's no way to stop it, and the people who try end up doing more harm than piracy ever did. The record industry's ridiculous holy war against file-sharers (more than 20,000 music fans sued and counting!) exemplifies the absurdity of trying to get the food-coloring out of the swimming pool. If the choice is between allowing copying or being a frothing bully lashing out at anything he can reach, I choose the former. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Cory Doctorow (2009). Little Brother. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/30142/pg30142-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

THE COPYRIGHT THING

A WAR-WEDDING

“What?!”, you say. “GraphQL is a server side query language. Redux is a client-side state management library. How could one replace the other?” Good question. Hold onto your butts, because I’m about to answer it. ⚛️ Switching to React First, a little back-story. Back in 2016, the front-end team at began switching our client-side code from a Backbone & Marionette stack to React. The declarative model for UI made much more sense than the MVC patterns we’d been dealing with. Pathwright It was a breath of fresh air and still is to a large degree. Everything was beautiful except for the state management side of our app. We turned to the Flux architecture quickly, and it felt like a major improvement at first. The concepts of circular data flow and single source of truth were philosophically sound and at least saner than the model-centric view binding approach of most MVC libraries. However, as our state management needs grew more complex, it started to feel like more and more layers of indirection. Once you have a handful of stores or branches in your state tree, you end up practically duplicating your server-side business data and relationships on the client. We had these beautifully declarative React components with a data layer that became a rats nest of actions, reducers, async middleware, and de-normed business data/logic. This all felt very wrong. ↪️ Switching to GraphQL That’s when we tried GraphQL. We began by implementing it on a new dashboard that combined a bunch of different data sources (this would have been a nightmare to do with our RESTfull APIs) and soon fell in love. It was like discovering React for the first time. Enthusiasm was high enough that we ended up getting our newly minted GraphQL server in production in only two weeks! Soon after, we started replacing a bunch of our REST APIs with GraphQL and it continued to be amazing. One of the side effects (sagas?) of this was that our UIs using these new GraphQL endpoints no longer needed stores at all. We started like we normally would with new stores, actions etc., but ended up deleting them because there simply wasn’t anything for them to do. 🤯 Three Startling Realizations This led to three startling, yet obvious in hind-sight, realizations for us: Most of our state management code was concerned with merging and mutating data from discrete REST resources into the right shape for our UI (reducers, selectors, actions etc.). A lot of our most complex state management was trying to manage the asynchronous nature of getting all that data in the right order for a specific route (sagas, middleware, thunks etc.). Practically everything else, UI state, worked great with plain old React state. “Damn”, we said, “damn,” while plunking coins into the swear jar. And then we deleted a of code. It felt good. lot So about GraphQL and Redux… My title was a little mis-leading (made you click?). What we really replaced was our REST API and then found as a result that most of our state management code was . no longer necessary When the client can control the exact shape of the state it needs from the server and get it all back in a single request, there’s simply not much need for state management libraries. To illustrate the point, let’s pretend our UI is having a conversation with our backend service through the state management library we’re using. Here’s what that might look like: So much of the work done by Redux and side-effect management libraries is trying to simplify that left-most conversation above. I would argue that for client-side apps, GraphQL can replace the need for Redux entirely. most I’m not saying Redux doesn’t serve a purpose. It’s a great library and the state management patterns it’s introduced are going to be around for a long time. Sometimes you don’t have control of your back-end stack and you’re forced to force REST to behave for your client-side UIs. That sucks and Redux really helps in that environment. There are also cases where you’re managing very complex state that needs trackable and consistent control: lower-level things like a client-side cache, offline syncing etc. Redux is great for these cases. In fact, some popular GraphQL libraries like Apollo can use Redux under the hood as a cache. BUT… . Switching will get rid of a huge amount of complexity in your client-side state management and reduce the scope of your client side code to just how data should render in the UI (which is what it should have been all along). If you can use GraphQL instead of REST, you should Oh, and you can still use Redux alongside of GraphQL, you just won’t need it very much. Plus you may be able to delete half your code which feels awesome. 🤔 Want to read more about how Major League Soccer had a similar experience? Great post on that here: https://dev-blog.apollodata.com/reducing-our-redux-code-with-react-apollo-5091b9de9c2a 👉Interested in trying GraphQL? Here’s a great place to get started: https://www.howtographql.com/ 🤨Thoughts? Questions? Rude comments? Hit me up on Twitter: https://twitter.com/wmdmark

Stories, News & Mentions about SPLUNKINC

Splunkinc’s Stories on HackerNoon

Splunkinc's News & Mentions