FiveTeams

fiveteams

OpenBB Podcast Transcript: Empowering Finance for All With Didier Lopes

The beginning of 2021 was challenging for our team, and as the story showed later, game-changing. At Metalamp, we started a partnership with , the company that founded the blockchain platform Cardano. Our team has been invited to take part in the Plutus Partners Program in order to help facilitate Cardano ecosystem development. IOHK IOHK compared the unused infrastructure they built with a new road with no cars. They were interested to check its quality and workability in the field. So, MetaLamp turned out to be one of the first vehicles that opened a new route. And its crew helped the builders of the road to search, find and fix all the «bumps». Willing to provide high fidelity testing, IOHK looked for the appropriate software development agencies all over the globe. After negotiations, only five teams were selected, and MetaLamp was among them. Why were we invited? There were four factors that let the MetaLamp team attract Cardano founders’ interest. 1. Solid experience in Haskell Since Plutus was written in Haskell, a profound knowledge of functional programming languages was a must-have. Thanks to involvement in the development of tricky and highly loaded projects with this technology stack since 2014, the team gained a rich experience in building complicated solutions in Haskell. If you have not used the Haskell language and the Nix development tool before, it is simple to drown in syntax and compilation errors learning Plutus. Kirill Elizarov, blockchain developer at MetaLamp 2. Ability to manage hard and tricky tasks and enthusiastic approach If we talk about the specifics of MetaLamp, we have always tried to take some non-standard projects, there have never been template ones. Every time we just study everything from scratch, think over the architecture, search for the information a lot, communicate with other experts who have experience in this field. Any project we create has the same specifics as working in an experimental mode. Most of the time is spent thinking through solutions, not writing code. Darya Trifonova, project manager at MetaLamp So, by the time of participation in the Plutus Partners Program the MetaLamp developers were ready to face any challenges: they used to work with unconventional tasks. We’re working with companies that really want to try something new, something innovative and interesting. Gerard Moroney, director of operations in IOHK And that is where the desire of MetaLamp to be ahead of the curve coincided with the opportunity offered by the Plutus Partner Program to try the unknown. 3. Proven track in blockchain By the time the Plutus Partners Program started, MetaLamp had been dealing with commercial blockchain products for five years. The team has experience with financial technology and has built applications involving lenders, loans, and ledgers. MetaLamp also performed blockchain projects related to different other domains: games, cloud services etc. 4. Special culture in the team MetaLamp is proud of the highly developed culture of self-education in the team. The company has its own educational program and developer development map that helps employees navigate the system of professional knowledge and skills. The development map gives us a very large fundamental base and traditions in self-development. In the absence of a culture of self-development, a culture of leveling skills, the implementation of the study of Plutus would take much longer, and it would be a more vague task for us than it eventually was. Roman Shtykh, CEO at MetaLamp Overcoming the training program and internal exams designed by the studio, IT engineers learn to fill the gaps in their knowledge quickly and on their own. As this ability to acquire necessary skills is honed sharply, developers can manage the tasks that require learning a lot of new things qualitatively and fast. Which tasks did we solve and what difficulties did we face? «If you are not embarrassed by the first version of your product, you’ve launched too late», Reid Hoffman, the founder of LinkedIn said. The team of Cardano developers pursued the aim to deliver the product on time, but first make sure its quality is worth recognition. That’s why they focused a lot of effort to attract heavyweight expertise in order to check Plutus’ workability. A smart contract is an application that constructs transactions and sends them to the blockchain according to definite rules. These rules cannot be violated by any blockchain user, they act like a contract in the traditional banking sector. The Plutus platform allows you to write smart contracts on the Cardano blockchain, which uses the UTXO model for ledger, which distinguishes it from other blockchains that have functionality for developing smart contracts. Stanislav Zhdanovich, blockchain developer at MetaLamp In this journey towards and the start of smart contracts on Cardano, the IOHK engineers wanted to feel assured that the next hard fork would match their high standards. deployment of Alonzo on the mainnet To complete the quality assurance tasks at a top level, the MetaLamp team members had to dive into the project and meticulously learn the required information. We were given time to study the documentation and lectures of the Plutus Pioneer Program before we started writing smart contracts. It was a bit difficult to understand, but the lectures helped a lot to “touch the code with your hands”. We could run it and see the result, make some changes of our own and see what happens. This experience tangibly helped me. This is a hard fork that includes smart contracts in the Cardano testnet. Testing made it possible to see in what form UTXOs are stored, how to use them in a transaction, and how to perform validation. Olga Klimenko, the MetaLamp blockchain developer One of the first obstacles the MetaLamp engineers faced was the lack of documentation to sort out some things. The only drawback of the program is that IOG is now actively rolling out new features and updates, sometimes even changing the main API of some functions, that’s why lectures can quickly become outdated. But the authors of the program are trying to keep it updated. By the moment, three versions of these lectures have already been published. Kirill Elizarov, blockchain developer at MetaLamp They worked in such a format when there were constant calls, questions about what challenges they faced, there was no such thing during the regular development process. There was a lot of support from the core internal IOHK team. We directly communicated with IOHK leaders, their operating officer and education director. It was super! Lars was very responsive and helpful (Lars Brünjes — education director at IOHK, PhD in pure mathematics passionate about functional programming in general and Haskell in particular.) Darya Trifonova, project manager at Metalamp The special community in Discord, where help can be asked, was very supportive for novice developers of smart contracts too. But sometimes we had to look at the source code of Plutus and figure it out ourselves Olga Klimenko, blockchain developer at MetaLamp She notes that fundamental preparation is a rewarding approach in any domain, and blockchain isn’t an exception. If you try to write smart contracts right away, it’s almost impossible. But if you start with studying the theory (what it is and how it works), then get over to the specific blockchain (in our case, Cardano), after that try to conduct simple transactions, and finally write smart contracts, then it becomes easier. In some cases learning the existing solutions in Ethereum served as a hint in a development process. Stanislav Zhdanovich, blockchain developer at MetaLamp During the period of MetaLamp participation in the program, IOHK did not have a testnet, which supported smart contracts.That also added some difficulties to the process. We only worked with the simulation, and when the testnet appeared, it turned out that it did not support many features, for example, PAB with wallets, state machines, etc. And the code that we successfully ran in the simulation would not necessarily work in a real network. It was difficult when we had to look for workarounds to make the contract work in the testnet, if some bugs did not allow it. We left reviews and talked with the Plutus developers and other teams who participated in the testing contracts program in order to discuss ways to fix or work around the errors. Olga Klimenko, blockchain engineer at MetaLamp Failures are the stepping-stones to success, as the proverb goes. And working on the project, MetaLamp got solid experience with Cardano tools. The profound expertise of our teammates was appreciated by IOHK engineers, and we were invited to take part as supervisors in the Cardano hackathon, based on the Plutus language. At Cardano Summit our developers helped participants write a defi application. What are the results that we have achieved in the Program? As a result, before the end of the program, we managed to write an NFT marketplace with an auction, but we did not have enough time to put it into the testnet. Rewriting with state machines took a long time, and the testnet met us with unexpected errors. When we launched a marketplace in the testnet, it turned out that there was no way to sign transactions, because there was no integration of wallets and PAB. Olga Klimenko, blockchain developer at MetaLamp By the moment, most of the problems that appeared during the end of the program have already been solved, so everything should go more smoothly in the testnet. Working in close collaboration with the internal IOHK team, the MetaLamp developers helped them find the bugs and participated in brainstorming on how it could be fixed or improved. To look wider, this experience allowed the development team to expand their knowledge significantly and start creating products not only on Cardano, but on Ethereum too. How has this experience impacted the MetaLamp expertise? Participation in the partner program was a turning point in the company’s work: the IT engineers dived into specific areas of the software development on blockchain platforms, acquired deep knowledge of Cardano and Ethereum and specifically, became the first developers who managed to write smart contracts on Plutus. We have become more well-known in the crypto community, and our team joined new blockchain projects. We began to expand the list of technologies, and now we are working not only with Cardano, but also with Ethereum. We have been invited more often as experts in the media and to various profile events Rostislava Glebovich, Head of sales at MetaLamp All this activity resulted in increasing the number of customers interested in blockchain technologies. At the moment we are involved in several ambitious blockchain projects related to DeFi. But since they are under NDA, we can’t share details. As we were the one of the first companies that started development of complex products on blockchain, we got an advantage in terms of better experience and proven expertise in the domain. It couldn’t help impacting the demand for our specialists and recognition of their skills in the community. Roman Shtykh, CEO at MetaLamp The MetaLamp team is excited by Cardano opportunities.The throughput of 257 transactions per second is a breakthrough in comparison with Ethereum that offers 8 times fewer or Bitcoin — 35 times fewer. The cost of transactions is lower than in other blockchain that also adds attractiveness to the platform. Energy consumption is optimized due to the consensus protocol, the Proof of Stake, which is obviously more ecology-friendly than the Proof of work. In the light of developing secure smart contracts, the high speed and low cost of transactions, and the ability for scaling make Cardano a promising technology for innovative projects. For example, MetaLamp team has been cooperating with Iagon engineers working on the decentralized Cloud Service. Having numerous awards in background and strategic partnership with IOHK, MetaLamp became «a perfect match» for an ambitious project. «Haskell developers are hard to come by, they are like diamonds, but worth more», says the client, assessing the experience of collaboration with our experts. Now smart contracts are a pain point in Cardano. But the developers work on the issues at a rapid pace. And when smart contracts appear in the mainnet, the popularity of the technology will skyrocket predictably. Many DeFi products will get funded, and as a result, demand for experts in the industry will increase. DeFi applications are developing by leaps and bounds, the market is growing every year, and it will grow further, and any consensus of economists and analysts says that this market will grow in the next 5–10–15 years. DeFi products have great potential, and they are taking a large share from traditional finances and will continue to take it away. To what limits they will extend, we do not know, but the potential of the market feels quite enormous Roman Shtykh, CEO at MetaLamp The company is happy to be at the origins of the creation of significant blockchain products and contributes a lot to ecosystem development. At the moment the team takes part in the development process of many significant projects based on blockchain, ranging from NFT marketplaces to heavyweight infrastructure projects, including Bridge, L2 etc. We’re proud that we’ve managed to overcome all the difficulties related to mastering a new technology, and at the new level we’ve become sufficiently equipped with knowledge and skills to meet any challenges that the market can offer.

How we Tested Plutus Smart Contracts on Cardano 

The leadership I’ve learned through life could be summed up and put on the front of a t-shirt, “ .” If that saying isn't already a thing: ™️. Though I left the U.S. Army a little over 20 years ago, I still stand by that statement. 95% of the leadership skills I learned, I learned in the Army : Most of my adult life has been spent in the military, until a year ago. I joined the Army in 1994, at 17 (my father signed the papers). In the end, I got bored training to go to war (I was young/dumb) and decided to get out four years later. Less than a year after that, I was back in the service, this time joining the U.S. Coast Guard. An obligatory backstory The time spent in the Coast Guard was split as both Enlisted and 15+ years as an Officer. However, given all of the training, schooling, and discipline I received along the way, I still credit my four years in the Army as setting the leadership bar for me. Here’s what I learned about leadership while serving in the Army’s 1/75th Ranger Battalion as an Infantryman, and how it’s applied to my life today, as a Project and Program Manager. Basic Training I thought this was the hardest thing I’d ever done in my 18 years of existence. There was very little in the way of leadership learned while attending Basic Training. Or, so I thought. Airborne School I thought was the hardest thing I’d ever done in my 18 years of existence. Learning how to jump out of a perfectly good airplane isn’t so hard once you do it. It’s the two and a half weeks getting to that point that’s hard. It’s full of instruction that’s ingrained into your head with repetitive motion, memorization, and the fear of possible death being removed via the squelch that was provided via continual movement and little real rest. Again, I didn’t see the application of any leadership education at the time. It was seemingly just a matter of staying alive as you walked out of a metal tube at 120+ miles per hour into the void. this RIP I thought I’d ever done in my 18 years of existence. I got my first glimpse of leadership while attending RIP when the cadre put the 30 hopeful guys into five teams with nothing more than a pile of gear and instructions to be out front in 30 minutes. Add to this an instructor walking in every three minutes telling us to do pushups or wall-sits, and you begin to see people crack from confusion alone. But wait, was that leadership? this was the hardest thing Ranger Battalion . My life at Battalion consisted of the following routine: run/lift/exercise, shoot, learn to run, lift, and shoot, run obstacle courses, maintain gear, learn close quarter tactics, hand to hand combat, and play hard. This was the gist of my everyday existence and a significant point of hardship for some. You could sum it up as an additional 3.5 years of Basic Training. But it was for a reason. I was wrong about how hard everything had been up to this point really The application of 4-years of leadership 👆 to Project Management Teamwork While I’m sure the mere mention of someone being in the military will conjure thoughts of collaboration, believe me when I say, it’s not the same across the board. Basic training proved this (teamwork?) in the first few weeks. However, the term and practice of teamwork was in everything we did at Ranger Batt. From entering a room and firing upon realistic targets to entering a bar in downtown Savannah. A Ranger would rarely be found alone, as the idiom state, “There’s safety in numbers.” While I wouldn’t be too far from calling a few of those men my friends, I’d be on the nose in referring to them as my teammates. Be it breaching a wall or contorting each other into submission during Jujitsu lessons; you came out knowing that the three for four nearest people to you at any given moment would be there for you when the shit got real. Amid a project, as the leader– inferred or otherwise– it’s your goal to build this kind of cohesion. I’m not suggesting you take the team to a ropes course or learn Jujitsu as a group (though it might help), but you need to take the time to figure out how to bring the team together. This will be the single most challenging thing to master, 95% of the time. Add in the current state of the world, and you being unceremoniously designated a Digital Project Manager, and you can raise that bar to 99%. Self Governing Over the last few days, while putting this article together, I didn’t initially have “self-governing” in the mix. I oft use the phrase “keep moving forward,” the source of which is my former squad leader yelling as we moved to another location. Now, remove that squad leader (he’s hurt, can no longer lead), or put it into a business context, remove your manager who didn’t show up to work. That team still needs to meet its objective, and that presentation you two were working on still needs to be presented to the executives– it’s not going to present itself. This is where self-governance comes in—taking the lead when there’s an unexpected lack of leadership. it It needn’t be anything so grave as a loss of life or one’s boss; there are plenty of times when you can step up to the plate and move the group, meeting, project forward. Learn to lead when there’s a lack of leadership, and your ability to build team cohesion when it’s needed most will come without even trying. [Could also apply to “picking up the slack.”] Idleness While being idle at home can be relaxing, it wasn’t “a thing” while in my barracks– it’s a concept that is "unlearned." If it was a working day, you were learning, doing, or otherwise remaining engaged during working hours. If there was “nothing to do” (<- I laugh writing that even today), you found something to do. Or, as your parents would tell you, they’d find something for you to do. In Airborne School, you're encouraged to learn on your own; it may lessen your chances of dying or, at the very least, getting hurt. This is applied to the office (or the home office), too; if your work/project is on hold or you finished your work for that day, use the time you have, the time you’re getting paid for, to remain engaged. Myself, I do training. And I make sure everyone knows about it. Your company expects you to keep abreast of the latest information concerning your job function. For me, that means learning some new ways to run a meeting or project management technique. Case in point, I’m working on my PMI-ACP (Agile Certified Practioner) training and application right now, next is to become a Certified Scrum Master. Why? Because I want to, and our bosses want to brag about us to their peers. That means we can’t stay idle; we need to keep learning, reading, and doing (read: “keep moving forward”). Time Management This is a huge one, perhaps the single most valuable “military to civilian” transitional skill taught/learned. While in the Army, or any job really, you’re given a list of objectives (think “needs to be accomplished”), and you’re usually given a specific time frame. If you’ve just destroyed the enemy bridge, yet you didn’t make it to the landing zone (LZ) for extraction in time, well, you might have a long, if not dangerous, walk ahead of you. The same, to a point, is correct in business too. If you have a task that you’ve been asked to complete and said you would, you need to get it done on time or risk the wrath of the enemy, 'er, I mean boss. My life now differs from then because I’m able to tell my manager that I under/overestimated my timeline, or I was outright wrong. I suspect most can do this without fear of reprisal (at least the first time). However, even this simple act is hard for some. Speaking from experience, it’s best to let your boss, manager, and team know that something went wrong (perhaps you misjudged something), and you cannot get the job done on time. Lying or just blowing off the deadline won’t fair well will everyone. Motivation Getting up in the morning sucks. You’d think being a career military man and a father of five, that I’d be able to get up without issue. If there’s an emergency, I can snap to-it without issue. But having the motivation to get up some mornings to work can be downright hard, especially now when I don’t have to see anyone as I’m working from home. Coffee does help. ☕️ I’m not going to tell you that being motivated to do everything/anything is easy, it’s not, and it’s a very personal thing. Sometimes people don’t like their jobs, and that can be a huge motivation drainer. However, if you have people counting on you, you need to pull up your pants, get a large cup of coffee/tea/water, and do your job. This will take motivation. You’re part of a team; you need to dig down and figure out how to get the job done. While it may be difficult for you to motivate yourself from the leadership side, others count on you to remain motivated for their sake. My wife would hear co-workers/subordinates say things like “his motivation is contagious” or something along that line (I’m not bosting, I’m making a point). Yet, she’d wonder why I wasn’t like this at home all the time. While it’s not a secret from her, sometimes, putting on that front for others is what gets the job done. Again, others count on you as a leader to motivate them. Do your team a solid and do just that. Persuasion It wasn’t until I was out of the Army some 5+ years that I realized I’d learned persuasive leadership. When I went to Officer Candidate School for the Coast Guard, I was with a group of guys cleaning a bathroom. Yes, even as a prospective officer, I got the lucky deed of cleaning the bathroom (or head if you will) 🚽. I’m not saying I am or was a good leader, but I persuaded the team of six that we needed to work together for the common goal of just getting it done. No big song or dance. It was the truth, I spoke it, and we got it done. I give my Platoon Sergeant, Sergeant First Class Dodson, the credit for teaching me this. He and I didn’t like each other much, but I was young, he wasn’t. He had a knack for motivating the platoon, but it wasn’t via motivational speeches per se; he did it by being truthful and making us all feel as if we were the ones who wanted to get the job done. He knew how to do his job; that’s why he was the man in charge. Wow, I really do want to walk 40 miles through the woods and mud with 110 pounds on my back. This goes smack in the middle of motivation and teamwork. It’s an art that takes years to perfect (I’m still working on it). Yet, if you’re able to do so, even a little, you can get the job done with even an unlikely team. However, the single “gotcha” that I can think of for this is the remote work scenario that many of us still face today. Trying to persuade someone to make the extra effort to get the project done is a lot harder when they can’t see or feel your energy or desperation. Clarity This is something I learned while just doing my job. That is, as a team, get from point-A to point-E without being seen, getting your Geometry homework done, and crocheting a sock. The point is, through all the stuff you need to get done, no matter how hard it seems, you need to take a moment to step back and look at the issue from the outside to see how this could be better… to have a moment of clarity. In the above case, separate the three tasks, and prioritize them. It’s not easy, yet you/we need to learn how to do this. We’ll call this “an adult thing.” I still fail at this one if I’m in the thick of it. I get too focused. It’s not until I’m done with work for the day that I see the error of my ways, that is, having that “duh” moment. And finally, Understanding willingness vs. ability This last lesson is one that anyone who joins the service or a first responder job can tell you they’ve learned. There’s a big difference between wanting/willing to do a task versus being unable to do it. Ethics aside, you can do a lot more than you think you can. You will do more than you think you can when the time comes. Before joining the Army, I wasn’t willing to jump out of a plane, ready to kill someone or march 35 miles in shitty boots. However, after my first year, I had jumped some 20+ times (almost) without issue, and every workday, I was learning how to kill people at the shooting range or hand to hand skills (it’s not as dramatic as it reads). As far as the road-marches, well, I just did those due to "external motivation." In business, while you’re not training to kill people, you will come upon those times when you don’t want to do something, but you’re able to. The job that was due this morning didn’t get done ( ), it’s Friday afternoon, and you’re the one who needs to do it to ensure the deed is done (it sucks being a leader sometimes, ‘eh?). You need to call your significant other and let them know you’re going to be late because you’re able to do this. This will rear it’s ugly head time and time again. It matters not though; if the task isn’t done, and you’re the leader, you’re the one that gets it done or take the hit. Choices... choices... Time Management This is the end 🤔 My point of sharing this is to show you that while we’re all learning things along the way, we might not know we’ve done so until the path worn by time is profound. You’ll have those ah-ha moments 💡 later in life and realize that you know how to handle this because your father’s friend’s neighbor taught you some 15 years ago. Just remember, keep moving forward, it’ll come to you. If you enjoyed this, you might also enjoy The Stakeholder Report , a weekly newsletter on leadership, education, business acumen, and updates to the global craft of Project Management.

What Being an Army Ranger Taught Me About Being a Project Manager

Earlier this year, Cold Iron Studios announced a new third-person survival shooter video game based on the Aliens film franchise called Aliens: Fireteam Elite. Cold Iron Studios has now that the game is going to launch on August 24 as a multi-platform release. revealed Along with the release date, Cold Iron Studios also revealed a new pre-order trailer showcasing the bonus items for the upcoming game. Any players who pre-order any of the slated editions for Fireteam Elite will receive the Hardened Marine Pack. This pre-order DLC offers players the Bandana Head Accessory, Digital Red Weapon Camo, 3 Weapon Decals and a Chestburster Emote. Obviously, the Bandana Head Accessory is an homage to the Vasquez character from the 1986 classic film written and directed by James Cameron. The Weapon Decals offer some new writing and images for the player weapons, such as "Adios" and "Endeavor. Aliens: Fireteam Elite Release Date Set for August 24th Aliens: Fireteam Elite is set 23 years after the original Alien trilogy. Players are dropped into the role of a Colonial Marine, who is aboard the USS Endeavor and answering a distress call to the outer colonies. From there, players head to various locations to battle the Xenomorphs across four campaigns with two friends or AI teammates.  Besides the Xenomorphs, players will also face Weyland-Yutani Synthetics. Fireteam Elite will feature five character classes, a vast array of customizable weapons and mods, character progression, advancement systems and more. Another feature is the Challenge Card system, so players will be able to alter each mission to give each playthrough a new experience. Overall, it's nice to see a new Aliens video game. Hopefully, it will fare better than the fate of the controversial Aliens: Colonial Marines, which was released back in 2013. Aliens: Fireteam Elite hits the shelves on August 24. The game will be available for Xbox Series X/S, Xbox One, PlayStation 4, PlayStation 5 and Windows PC via Steam. The new pre-order trailer is viewable below. More in Gaming: What is the Sony PlayStation 5 System Software Beta Program? Cyberpunk 2077 Returning to PlayStation Store on June 21 The Legend of Zelda: Breath of the Wild Sequel Set for 2022 Launch Keep up with all the latest gaming news, features, and guides. Subscribe to our newsletter in the footer below!

Aliens: Fireteam Set for August Launch, Pre-Order Bonuses Revealed

It looks like the Alien franchise will be getting another shot in the gaming realm. this week unveiled a new AAA game based on the Aliens franchise. Titled Aliens: Fireteam, the new game will be a co-op third-person survival shooter set in the Aliens universe, and it’s currently scheduled for a Summer 2021, multi-platform release. Cold Iron Studios Cold Iron Studios is developing Aliens: Fireteam in conjunction with 20th Century Games. It will be available for Xbox Series X/S, Xbox One, PlayStation 5, PlayStation 4, and Windows PC via Steam. Players will assume the role of a Colonial Marines squad as they are matched up against hordes of Xenomorphs that will swarm over a given area. Cold Iron Studios Head Craig Zinkievich is promising that the game will offer character customization, tons of weapon options, and gear in order to survive the Xenomorph onslaught. Fireteam is set 23 years after the events of the original Aliens trilogy. A team of Colonial Marines on the USS Endeavor have been charged to answer a distress call on the outer colonies. The game will feature multiple maps in overrun facilities, abandoned ruins, and alien landscapes over four campaigns. Players will team up with two friends or AI teammates. The game will also feature up to 20 enemy types ranging from the Xenomorphs to Weyland-Yutani androids. The last decent video game based on the alien series was Alien: Isolation released in 2014, which was more in line with the original 1979 Ridley Scott movie. This game is taking more of its cues from James Cameron’s 1986 Aliens film. The last video game to do this was the greatly maligned, highly controversial Aliens: Colonial Marines, released all the way back in 2013. Besides the customizable weapons and mods system, Fireteam is set to feature five character classes along with character progression and advancement systems. The Challenge Card System will enable players to alter each mission to give them a unique experience on every playthrough. Thus far, the Aliens: Fireteam trailer looks promising, and it has authentic sound effects from the film. Hopefully, the release goes smoothly. You can get a look at the game’s reveal trailer here: More Gaming News Pokémon Legends: Arceus Revealed Alongside Diamond and Pearl Remakes Ninja Gaiden: Master Collection Arrives for All Platforms on June 10 7 Nioh 2 Character Creation Codes on PC

Aliens: Fireteam Arrives This Summer with Third-Person Co-op Shooter Gameplay

In 2001, Microsoft launched Halo: Combat evolved alongside its tie-in novel , authored by Eric Nylund, who then wrote in 2003 and in 2006. Since then, nine other authors have contributed novels to the series, adding more lore to the ever-expanding universe of Halo. One of these authors is Troy Denning, who wrote works such as and The Fall of Reach First Strike Ghosts of Onyx Halo: Last Light, Halo: Retribution, Halo: Oblivion, Halo: Shadows of Reach. About Troy Denning is an American fantasy and author and game designer who has written over two dozen novels, including , and . In this exclusive interview, Troy sheds light on how he became an author, his past projects as a game designer, and some exciting things to look forward to in his latest book, . Troy Denning science fiction The New Jedi Order: Star by Star Waterdeep Halo: Outcasts When did you start writing stories? I think it was in the eighth grade, so probably about the age of 13-14. Our English teacher assigned us to keep a journal, but writing daily entries about chores and ski team practice seemed boring. So I just started to write stories instead. What made you want to start writing? If it was a book, what was it? I guess I became interested in writing because I was an avid reader from a young age.  I would get sucked into worlds created by Asimov, Heinlein, Tolkien, Burroughs and began to imagine my own versions. At first, my worlds were pretty derivative of whatever I happened to be reading, but as my archive expanded, so did my imagination. It wasn’t long—maybe three or four years—before the worlds in my head began to seem fairly original (at least to me), and I started to feel like I had something worth putting on paper. You know, I would read something by Clarke or Leiber and think “I could do that!” Of course, I couldn’t. The mark of a true master is making their work look effortless, and that was certainly true of the authors I was reading. It took fifteen years, a B.A. in English, and eight years of experience as an editor and tabletop game designer before I published my first novel. What does your writing process look like? What kind of routine do you have, and how does it help you? Two of my favorite books on writing are by Terry Brooks, and On Writing by Stephen King. Both authors are (obviously) masters of the craft, but about the only thing their processes have in common is they write every day for AT LEAST a few hours.  Otherwise, their processes are almost opposites. King is a wanderer. He just starts writing and sees where the story takes him. Brooks is a mapper. Before he starts, he likes to know where the story is going and how he’s going to get there. Sometimes the Magic Works When I started out, I was more like King. I just started and let the story take me where it wanted. It’s incredibly fun, but you end up wandering down a lot of box canyons and having to backtrack to keep the story moving forward. If you have the time and self-confidence to handle the bad turns, it can produce some truly marvelous results. That’s how I wrote the Prism Pentad (Dark Sun) and Pages of Pain (Planescape).  But I also have a drawer full of unfinished books that I couldn’t figure out how to fix. More importantly, when I shifted from working with people who knew me personally (as the editors for the Dungeons & Dragons stuff did) to people who were contacting me to write for licensed Intellectual Properties (IPs) such as Star Wars and Halo, the process needed to be a lot more structured. Those books are part of a major publishing program that ties into other mediums such as movies, TV, games, comics, etc. There are usually well over a dozen decision makers (publishing house editors, continuity editors from the IP, executives, marketing people). They can’t just say, “write a book featuring the Lady of Pain” (as was the case for Pages of Pain) or “give us a five-book series set in that game world you just designed” (as was the case with the Dark Sun books).  Everybody needs to agree on the story in advance, and that requires detailed outlines. So now, I work more like Brooks does. I construct detailed outlines before I write the first word of the first chapter. All of the editors and executives and sometimes even the marketing people make suggestions and offer notes, until we finally have something we all agree on. By the time we’re finished, I have a very clear picture of the story arc, the character arcs, the settings, and most of the other story elements. That isn’t to say that nothing ever varies from the outline, because it often does. But when that happens, we have a common map, and we can all agree on the changes I need/want to make. (Or do something even better, which often ends up being the case after everyone offers input). What would you say is the hardest part of writing a novel? For novice/young writers, finishing what you start is always the hardest part. They lose confidence in what they’re doing, because they haven’t been through the revision process and don’t understand that the first draft of any novel is just raw ore. It takes time and effort—a lot of both—to run that draft through the crusher and flotation tanks and smelt out the gold. For me, the opening is always the hardest part. It’s where you raise the story question, introduce the characters, establish atmosphere, and set tone. The first chapter of a good novel does an incredible amount of work, and by the time I sit down to write it, I have a clear concept of how ALL of that stuff is going to work. But getting the concept onto the page? That can be a struggle. It’s not unusual for me to spend a month or more just getting a first chapter I’m happy with. But once that foundation is laid . . . I wouldn’t say the rest of the book writes itself, but it’s a lot easier. Throughout your career, you’ve written several books for various sci-fi and fantasy settings. Which universe did you enjoy writing for the most and why? What was your favorite book in that series? Boy, that’s a loaded question . . . and one I can’t answer simply. Writing a book takes a LOT of time and energy, and if you’re just doing it to fulfill an assignment, you’re doing it wrong. If it’s just work, you can find a lot of jobs that reward you better for your time. The truth is, I loved all the properties I wrote for, or I wouldn’t have done it. I went to work for TSR Hobbies straight out of college because I loved Dungeons & Dragons, and in addition to the work I did as an editor and game designer, I ended up writing twenty novels for them.  Eventually, the opportunity to work on something I loved even more—Star Wars—came along, and I wrote twelve novels for them. Then Disney bought Lucasfilm and I didn’t love where the line was going as much, but I’d started to explore the Halo universe and was really falling in love with it. So when Ed Schlesinger called to see if I’d be interested in doing a Halo novel, it was an easy yes. In addition to your novels, you’ve also worked as a game designer. What past projects are you most proud of? I would have to say the Dark Sun setting, which I conceptualized with Timothy Brown, Mary Kirchoff, and Brom, and wrote with Timothy Brown.  It’s a rough, gritty, campaign world set in an environmental wasteland (yes, it IS a warning about global warming and unchecked environmental damage), and it reflects my love of sand-and-sorcery fantasy like Conan. In Halo: Outcasts, we see the Sangheili society under Cortana’s rule. What was it like describing a society, which once prided itself on its culture of honor and strength, brought to heel by its new overlords? I probably have a different take on martial culture than a lot of readers. Any society that needs a code of honor to regulate its violent interactions is two things by definition:  a) violent, and b) probably not all that honorable. If there wasn’t a fair amount of violence, the society wouldn’t need to regulate it. If warriors were inherently “honorable,” there would be no need for a code. They would just naturally behave with honor. So, when I look at the Sangheili, I see a fairly violent species that, at some point in its past, was so ruthless it developed a code of honor to keep from tearing its society apart. Now, what happened to their ruthless streak? Did it just go away? Nature doesn’t work that way. The ruthless streak is still there. It’s just sublimated to an external code of honor. The thing about sublimation is it’s imperfect. Sublimated traits manifest themselves in other ways. And that’s how the Sangheili kaidons have been portrayed in lore. They’re scheming, sneaky warlords maneuvering to secure their own interests. Once they have the advantage, they usually find a way to justify any amount of underhanded violence in the name of “honor.”  So, when I look at Sangheili culture, I tend to see parallels with Feudal Japan, and the Arbiter’s struggles in unifying the Sangheili are very much akin to that of a feudal emperor fighting to control his shoguns. We are also introduced to an Oath Warden, Crei ‘Ayomuu. Can you tell our readers about Oath Wardens and their role in Sangheili culture? An Oath Warden is basically a bounty hunter who takes contracts to enforce agreements on behalf of other Sangheili. They are unique in Sangheili culture in not considering themselves bound by the warrior code of honor, which frees them to use any means necessary to fulfill their contracts. At the same time, they follow their own code, which they claim to be far more strict than that of the warriors. The thing is, only Oath Wardens know whether that’s true, since only Oath Wardens are allowed to know their code. Halo: Outcasts also focuses on Spartan Vale, a member of Fireteam Osiris from Halo 5. What was it like writing for a character that hasn’t gotten much attention since the events of Hunters in the Dark and Halo 5: Guardians? I enjoyed it. In various pieces of Vale’s background lore, there are hints of a complex, engaging character with the potential to play a much larger part in the Halo Universe than she has so far. I enjoyed building on those hints, to create someone who I hope the readers will want to see more of. What else can Halo fans look forward to in your latest book? We’ll revisit some characters from Halo Oblivion, learn a few of Netherop’s secrets, and explore some very ancient lore. You often write for pre-established universes (Halo, Star Wars, Dungeons and Dragons). Have you ever considered writing something in your own setting? If so, what would it look like? I have a drawer full of partially-completed spec projects in various settings. There are sand-and-sorcery fantasies, contemporary thrillers, and near-future SF in there. Fortunately/unfortunately (it’s a bit of both), the media tie-in contracts have kept me too busy to complete them. When an offer for a cool project that will definitely be published comes in, the cool project that might be published loses out. Sigh . . . maybe when I retire! Do you have any advice for aspiring writers? Sure: 1) write every day; 2) write because you love it, not because you want to “be” a writer; 3) don’t count on making a living as a writer; only a very small percentage of us do, and the percentage who make a good living is even smaller; 4) study your craft; read books like On Writing by Stephen King, Sometimes the Magic Works by Terry Brooks, Story by Robert McKee . . . really, anything about writing that you can get your hands on; 5) editors are your friends; even if you’re not ready to publish yet, hiring an independent story editor to mark up a short story or a chapter of a work-in-progress will teach you a LOT about writing.

An Exclusive Interview w/ Troy Denning, Author of Halo: Outcasts

Una entrevista exclusiva con Troy Denning, autor de Halo: Outcasts

Une interview exclusive avec Troy Denning, auteur de Halo : Outcasts

हेलो: आउटकास्ट्स के लेखक ट्रॉय डेनिंग के साथ एक विशेष साक्षात्कार

『Halo: Outcasts』著者、トロイ・デニング氏への独占インタビュー

Uma entrevista exclusiva com Troy Denning, autor de Halo: Outcasts

Một cuộc phỏng vấn độc quyền với Troy Denning, Tác giả của Halo: Outcasts

《光环：流放者》作者 Troy Denning 独家专访

During the height of the business intelligence (BI) craze earlier in my career, I worked with an internal reporting team to expose data for (ETL) processes that leveraged data structures inspired by . It was a new and exciting time in my life to understand how to optimize data for reporting and analysis. Honestly, the schema looked upside down to me, based on my experience with transaction-driven designs. extract, transform, and load Ralph Kimball In the end, there were many moving parts and even some dependencies for the existence of a flat file to make sure everything worked properly. The reports ran quickly, but one key factor always bothered me: I was always looking at yesterday’s data. Due to the increase in size, velocity, and types of data being produced in modern tech stacks, real-time data has become the holy grail for enabling up-to-date and proactive analysis and data-driven business decisions. When we use the common batch style approach to data ingestion, the data is always stale, and it can’t answer the pressing questions. To enable these real-time use cases, the data must be available in the storage layer and processing layers currently implemented at an organization. For many years—and this is still true today—transactional (OLTP) systems like Oracle, Postgres, and SQLServer are at the core of most businesses, and they’re central to downstream analytics and processing. This work is typically done by a team of data engineers who spend months building pipelines that pull and transform data from those OLTP systems into analytics data lakes and warehouses (OLAP). Over the past decade, organizations of all sizes have spent billions of dollars with the goal of improving their capacity to process data and extract business value from it. However, only of organizations believe they have achieved the vision of becoming a data-driven company. One of their biggest hurdles to achieving the “Data Utopia” state is gaining the ability to build reliable data pipelines that extract data from OLTP systems while ensuring data consistency and ease of scale as the data grows. 26.5% This made me wonder if there was a better way to accomplish the same results without a hefty overnight process. My first thought was to explore the change data capture (CDC) pattern to meet those needs. Why Change Data Capture (CDC) Matters Extracting data from OLTP systems can be cumbersome, for reasons that include: Unlike SaaS tools, enterprise databases don’t have flexible APIs that you can leverage to react to data changes. Building and maintaining pipelines to extract data from OLTP systems, especially as a business scale, is complex and costly to ensure data consistency. On the other hand, CDC is a software design pattern that determines and tracks database updates as they occur. This allows third-party consumers to react to database changes, vastly simplifying the process of retrieving data changes in the OLTP system. CDC events can be sourced through several different approaches: Time-based: uses the required timestamp column as a source for data Checksum: leverages checksum on tables in source/target, using differences as a source for data Log-based: reads database logs as a source for data For this publication, we will focus on the log approach. Log-based CDC is commonly accepted as the best option for real-time use cases because of its lightweight approach to producing data changes. Time-based and checksum approaches require more overhead and CPU, often using queries against the database to find the newest changes to a dataset. CDC using log-based reads can shortcut these expensive and time-consuming queries, capturing changes by going straight to the source: the logs. Using the log approach, database writes are handled via transaction logs, often referred to as changelogs. CDC solutions listen to those same logs and will generate events where established conditions apply. By using the logs, the CDC service gains access to the same data without waiting for the transaction to commit and then making a request to the database for that data. By implementing a CDC solution, third-party data consumers have quicker access to the database changes. Additionally, data consumers do not require access to the actual database since they are only interested in updates in those changelogs. Not only does CDC allow near-instant data replication, but it also has minimal impact on the performance of production databases. Because CDC only accesses the logs and doesn’t pose any security threat, the IT teams love it too. The unique characteristics of CDC made it widely adopted by many tech leaders that real-time use cases are key to their business models (e.g., , , ). Shopify Uber CapitalOne To understand CDC more, let’s consider a use case related to the financial industry. Detecting Fraud Using CDC Fraudulent transactions are a major concern for the financial industry. In fact, noted credit card fraud increased 44.7% in 2020 over prior-year values. this report Let’s assume WorldWide Bank, a fictional financial institution, wants to improve the speed at which they recognize fraud. Their current approach relies on an ETL process to query against their Oracle database and generate reports. They would like to explore using in order to process against the Oracle database changelogs to generate events that will feed their Databricks Delta Lake prototype. Like many organizations, WorldWide Bank has a variety of transactional systems, including a large subset of data in an Oracle database. They have a growing need to provide more machine learning (ML) capabilities as well as combine various datasets in an easy-to-use environment for analysis, which has led to their adoption of Databricks for a data lake. Arcion CDC The relational data model of Oracle, along with the hassle of bringing other ancillary data sources into Oracle and the lack of support to run ML directly on Oracle, makes Databricks an easy choice. But, this approach leads to another challenge: how to source the Oracle transactions into Databricks in a timely manner. To improve the speed at which they can detect fraud, it is critical to their operation to be able to source CDC events from Oracle quickly, depositing them into their Databricks data lake for further processing and detection. Regarding their decision to use Arcion CDC, why might WorldWideBank choose Arcion and not another option, even open source? The answers are simple. Debezium, for example, is an open-source option, but it requires the installation and running of Zookeeper, Kakfa, and MySQL—all before you can even begin to use or install the tool. This alone would require a large team of engineers to simply install and maintain the tech stack. While other tools—like Fivetran’s HVR for CDC—might also offer free trials, Arcion has two free options to choose from: a 30-day trial with a download or a 14-day trial on the cloud. This is convenient for testing, so it made Arcion an easy choice when selecting a CDC solution. At a high level, the Arcion CDC architecture can be illustrated as follows: If this approach goes well, WorldWide Bank will expand the design to replace other source databases in Oracle, Microsoft SQL Server, and MySQL. Prototyping Arcion CDC In order to simulate the WorldWide Bank scenario, let’s assume inbound transactions are being written to an Oracle database table called TRANSACTIONS, which has the following design: An example record in the TRANSACTIONS table would appear as shown below using JSON format: {
   "id" : "0d0d6c35-80d0-4303-b9d6-50f0af04beb3",
   "post_date" : 1501828517,
   "trans_date" : 1501828517,
   "type" : 1017,
   "card_number" : "4532394417264260",
   "account_number" : "001-234-56789",
   "amount" : 27.67,
   "vendor_number" : 20150307,
   "vendor_location_latitude" : 42.3465698,
   "vendor_location_longitude" : -71.0895284,
   "vendor_name" : "Berklee College of Music Bookstore" 
} In order to leverage the power of Databricks and Delta Lake for fraud prevention, we need to set up a CDC service to act as middleware between the two systems. The TRANSACTIONS table design and data in Oracle need to exist as a Bronze (Ingestion) layer within Delta Lake. Once the existing data has been ingested, all future transactions will leverage the CDC stream included in the Arcion solution. ** **Getting started with CDC begins with establishing a free and obtaining the connection/login information for the Oracle database and Databricks system. Arcion Cloud account After gathering this information, I started a new replication in the Arcion Cloud application and defined the replication and write modes between the source and target database: In this case, I selected the replication mode. This lets me replicate a full snapshot of the database and enables real-time operations for all future updates. I chose the write mode option to fully replace any existing database design and data on the ta Full Replacing rget host. Next, I established a connection to the Oracle database, similar to what is shown below: We can validate the connection by using the Test Connection button. Once ready, the button moves the progress to the next step. Continue to Destination On the Destination screen, I can add a connection to the Databricks service and select it as the target for the replication being created: Using the button, I was able to proceed ahead to the filter screen shown below: Continue to Filter I selected the source database and tables on the Filter screen. Once ready, all I had to do was click the button to complete the following tasks: Start Replication Remove any existing databases in Databricks that match the current replication request Establish new databases in the Databricks environment Perform a snapshot replication of all existing data that matches the filters provided in the replication request Establish a CDC stream between Oracle’s changelogs and Databricks for all future transactions. I completed all of this with zero code. Once the replication started, I could monitor the process in the Arcion Cloud user interface: As noted above, the snapshot will be performed first, then any future updates will arrive via the CDC stream. Once completed, the user interface is updated as shown below: At this point, the Bronze (Ingestion) layer of Delta Lake is ready for use. The ingested data can be transformed into Silver (Redefined Tables) and Gold (Feature/Agg Datastore) layers, allowing us to leverage the benefits of Delta Lake functionality to detect and mitigate fraud. Going forward, the expectation will be that all data from the Oracle TRANSACTIONS table will automatically find its way into Databricks. We have one final step of our CDC ingestion into the data lake. Prior to running the ML fraud model for prediction output, we would run pre-built dbt tests or Great Expectations checkpoints to ensure we have received new records since our last fraud detection run. This would give WorldWide Bank confidence that full end-to-end CDC from Oracle is flowing into Databricks as expected. I find it very impressive that I can establish this level of design by merely following a series of steps within the Arcion Cloud user interface without writing a single line of program code. For the WorldWide Bank use case, establishing similar replications for their other databases should be just as easy, allowing all fraud detection to be processed in one single system. Conclusion Since 2021, I have been trying to live by the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality which extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester Arcion provides CDC functionality without reinventing the wheel. Their platform allows feature teams to utilize a no-code approach to establishing source and target data sources, plus everything needed to route inbound database transactions to a secondary source before they are written to the database. In addition, Arcion can be deployed in multiple ways. Financial or health care enterprises, for example, maybe have security or compliance requirements. They can deploy self-hosted Arcion on-prem or in a virtual private cloud. For companies without the DevOps resources to manage their own deployments, fully-managed Arcion Cloud would be a good option. Furthermore, the ease at which snapshot and CDC streaming replications can be established allows for additional data sources to leverage the same processing logic found in the Databricks Delta Lake implementation. This ultimately provides a “don’t repeat yourself” (DRY) approach by allowing one source of logic to focus on fraud detection. The Arcion team is certainly adhering to my personal mission statement and allowing their customers to remain focused on meeting corporate priorities. If I were working with that same BI team that I mentioned in the introduction, I would certainly recommend exploring Arcion Cloud to meet their needs. This would provide the opportunity for real-time data rather than relying on day-old data to make decisions. Have a really great day!

How To Use Change Data Capture for Fraud Detection

Last year we saw an explosion of startup investments in no-code and low-code data platforms as well as open-source projects in the data space, which primarily define the modern data stack. It is time to dive a little deeper into the topic and understand the dynamics in those markets. In this article, I highlight the reasons for these booms and would like to draw attention to the problems of the markets to ultimately express the thesis — Are the final days of the classic SaaS model in the data space counted? The complexity of data projects 8 years ago, companies started to look at how to make better decisions for the business using the data they collect. At that time, I started my career as a data scientist and I heard for the first time that there was a shortage of data scientists and software developers. Today in 2022, that picture has become even more dramatic. The demand for data and engineering jobs is at twice the rate of the absolute supply. Salaries for data talents exploded and data science was the “ twice in a row. And the demands for developers continue to grow: new frameworks, new apps, and new ideas require complex solutions. Two years ago in Rasa’s office in Kreuzberg Berlin, one of my favorite data bloggers told me that the “Data science fallout winter” is coming. What he meant was that companies were disappointed that data projects rarely end as a success. From my observations, this was due to several reasons: sexiest job in the world” in a way that translates business goals into a data strategy and data project. The feedback loops were taking too long. It takes a long time for a manager to align with a data scientist and software engineer A data project cannot be treated like a normal software project which is already fairly complex. Expectations were set too high and timelines were planned far too tightly. a lot of the resources. The data collection process and cleansing tied up Then, on a Venturebeat panel, the often-quoted figure was thrown into the room: . The symptom of the data science fallout winter was there. 85% of data projects fail The renaissance of open-source led by the modern data stack While companies were a bit timid about investing in data science and business intelligence projects, grandiose solutions were emerging among software developers. GitHub was the place where big solutions were born. A safe space for engineers with a vision. On GitHub, some of the hottest frameworks in the data space quietly emerged. We are talking about: (database), (data pipelines), (transformations), (data extracting), and most recently (data extracting). These tools are often summarized under the buzzword “ ”. They run flexibly on a data warehouse like Snowflake, the setup is relatively simple, and the services can be combined to map the processing and analysis of data. Elastic Search Airflow dbt Meltano Airbyte modern data stack But why open-source? Why hasn’t Microsoft or another tech giant owned and branded the modern data stack? The answer is trivial and the realization has been world-changing: Data projects are the most complex task in software development. Every data project is an edge case no matter if the business question might be the same. Data from a wide variety of sources with varying data quality rushes into enterprises at an even wider variety of frequencies. The data must be melted together, and the methods of analysis and interpretation of the data are as diverse as fauna and flora. Complex software can only be accomplished with many differently qualified developers. So many developers that even a tech giant would not have been able to build the foundation for the modern data stack without neglecting the core business. What did the modern data stack change for companies? In the meantime on the business side, more and more companies are adopting the modern data stack with a data warehouse in the center point of business intelligence (BI). I am very happy to follow this development since companies were able to work lean and agile with data. But one problem remains: Companies still have too few developers. And in data projects, managers and engineers still don’t speak the same language. The role of the business analyst got more and more relevant! The mediator between both parties: a BI Analyst that understands the fundamentals of data analysis and the objective of increasing the company’s ROI but lacks the technical coding skills. Startups emerged in 2021 with the promise of connecting data without coding skills, creating analytics, and moving companies forward in the digital transformation and data literacy. The startups build on the open modern data stack and give it a user-friendly wrapper. Prominently, they even advertise that the UI would replace dbt, Airbyte, Snowflake, and Airflow. However, the truth is that all of these technologies run in the background of the software, which is resold under their license. The modern data stack that was developed to be freely available is now being sold to customers as a SaaS tool? If you take the market cap of the closed SaaS solutions together and compare it on the time axis with the market cap of opens-source solutions, you can see how the market gains for SaaS are decreasing. On the other hand, a very steep growth can be seen for open-source solutions. Early open-source companies like RedHat, databricks or elastic are nowadays publicly traded companies. l puts it this way: . OSS Capita “ Open-source is eating software faster than software is eating the world ” Each use case must be built by the internal development team itself. This means that the focus can only be on simple, top use cases. Sales takes place via sales reps and conventional marketing. Scalability is limited. → High costs for sales and product development No adaptability of the tool to the individual use case. High lock-in effect. Training for new users required. → The low cost-benefit for the customer in the long run In the end, we see a frustrated customer who uses a tool because it was bought once but the results are not satisfying. This issue can be well illustrated by the example of Airbyte vs. Fivetran. Fivetran started in the market 10 years ago and helps software engineers to extract data from one source and load it into another data source, e.g., a data warehouse. Over time, many new data sources evolved from SaaS tools such as advertiser tools (Facebook), CRM, and reporting tools. Today, the number of SaaS tools exceeds 10,000, and with Fivetran’s closed approach, it was only possible to maintain and connect 1% of the data sources. And this process alone is a heavy lifting task if you build it internally. It is also very costly, which is also reflected in the subscription prices. As a result, customers were increasingly frustrated, paying a high price for limited usability. Airbyte recognized this problem 1.5 years ago. Instead of relying on a closed approach, Airbyte adopted an open-source business model. The Airbyte team makes it easy for developers to connect new data sources that are still missing. Airbyte is free to use and easy to set up by an engineer who finds the repo on GitHub. Thus, the complexity of the tool continues to grow independently with each new use case and data connector built by the community. Key capabilities for shift hand in hand with the different business models. It is now about serving the contributors in the community, making it easily accessible, and building out the overarching roadmap. Developing a convenient solution for enterprises that feels not like a ripoff. The success proves Airbyte right, within 1.5 years Airbyte won 16,000 customers while Fivetran stands at 2,000 customers. Of course, Fivetran customers are all paying, while Airbyte has a lower turnover, but considering the long-term development and renaissance of open-source, it is only a matter of time until enough customers pay for Airbyte. The popularity, the developer-friendly features, and the transparent subscription model is too convincing for that. (Picture Airbyte vs. Fivetran) The remaining problem with open-source The success story of Airbyte should not hide the problems of open-source. An average contributor works only 3 months on an open-source project before moving on to a new adventure. All the pressure is on the main contributors and initiators of the project. This problem became obvious with the Log4j vulnerability in 2021. Many large companies including Apple, Microsoft, and Cloudfare were using the open-source library. When the vulnerability became known, these companies turned to the Log4j team to fix the problem as soon as possible. We are talking about multi-billion dollar companies here, turning to a group of idealists, some of whom only work on the libary in their spare time. Understandably, the situation feels thankless, since most companies don’t pay for tech support or even make it known to them beforehand that they were using the Log4j libary to the initiators. The call became louder for more transparency and a system of monetization and valuation of the work. This is the biggest hurdle open-source software has to overcome. Contributors should feel appreciated and get something in return (dollars are also not always the answers). Systems should be developed that keep contributors longer on a project and value their work adequately. What do we learn from this? In summary, we are currently experiencing a renaissance of open-source software. Traditional SaaS business models in the data space are difficult to implement because data projects are too complex to solve in a proprietary tool for customers. I drew an example around no-code data platforms that rely on SaaS business models in particular but build on open-source without openly admitting it. This can lead to a problem in terms of customer satisfaction. That this is problematic is shown by the Log4j case as well as the Airbyte/Fivetran case study. Closed SaaS tools are not only expensive to build, but also do not address the complexity of customer problems. Because closed tools rely heavily on open-source libraries, non-transparent communication also creates major security vulnerabilities. There is a certain frustration and tension on the side of open-source contributors. This holds opportunities to change something, create completely new tools, business models, and software categories. And this is into what Kuwala turned now: an Open-Source No Code Data Platform. Give us a try on ! You have a different perspective? Or would like to discuss it more in depth? You can easily join our discussion on slack, . Or just comment below. Github here

We are Currently Experiencing a Renaissance of Open-Source Software

Halo stands today as one of the greatest gaming franchises of all time. It’s among the most well-known first person shooters out there, to the point where it’s even received a . With the recent release of Halo Infinite, it’s interesting to look back on the series and how its story was shaped over the years. That’s why this article will take a look at all Halo games in chronological order. live-action TV series This article will only consider the games and the events that take place during them. Novels, webseries, and in-game data pads will not be mentioned.The only game that won’t be included is Halo: Recruit, a short VR experience with no clear timeline indication. All Halo Games in Order Chronologically Year 2531 Halo Wars Year 2552 Halo: Reach Halo: Combat Evolved Halo: Fireteam Raven Halo 2 Halo 3: ODST Year 2552-2553 Halo 3 Year 2554 Halo: Spartan Assault Year 2557 Halo 4 Year 2552-2557 Halo: Spartan Strike Year 2558 Halo 5: Guardians Year 2559 Halo Wars 2 Year 2560 Halo Infinite Halo Wars - Year 2531 Set around 20 years before the first Halo game, Halo Wars features a Real-Time Strategy gameplay style shared by almost no other title in the series. Players command the human forces of the USNC during some of their earliest battles against the evil alien Covenant. This title is the first one to occur according to the series’ timeline, topping the list of Halo games in chronological order. Halo: Reach - Year 2552 Acting as a more direct prequel to Master Chief’s games,  Halo: Reach takes place mere weeks before Combat Evolved. The player joins a team of Spartans as they battle against the Covenant, struggling against impossible odds to turn the tide of their war. It’s also the only Halo game to feature space combat, making for one of the most unique experiences in the series. Halo: Combat Evolved Combat Evolved is the first-ever Halo game, as well as the first game to feature John-117 - also known as Master Chief - in the starring role. Combat Evolved defined the modern era of first-person shooters, providing intense gunplay and even a multiplayer mode. Its story was also surprisingly in-depth, featuring a new faction known as the Flood to cause chaos amongst humans and aliens. Some might consider it dated by present standards, but it’s a fantastic title to start your journey in playing all Halo games in order - be it by release date or by chronology. Halo: Fireteam Raven One of the more unique titles on this list, Halo: Fireteam Raven is an on-rails arcade game you won’t find at home. It’s set during the same time as Combat Evolved, with the eponymous Fireteam defending Installation 04 from Covenant forces. It’s not exactly an easy game to experience, due to its status as an arcade game, but it’s a perfectly serviceable addition to fans of the franchise. Halo 2 Set around one month after Combat Evolved, Halo 2 continues the battle between the Covenant and Flood forces. It also allows players to experience the campaign through the eyes of a Covenant soldier known as the Arbiter. On top of this, Halo 2 was the first game in the series to feature online play, a staple feature that would popularize the franchise to millions of fans. Halo 3: ODST Halo 3: ODST is a complicated title when it comes to playing the Halo games in order. It occurs during the events of Halo 2, but is meant to act as a more direct prequel to the next title in the series. Players take the role of a rookie, teaming up with ODST soldiers to fight against the Covenant in defense of Earth. Its gameplay is much more open-ended than previous entries, and ODST even came packaged with a copy of Halo 3’s online multiplayer. The title might not feature Master Chief in much capacity, but it’s a great experience that any Halo fan should try. Halo 3 - Year 2552-2553 With the tagline of “Finish the fight”, Halo 3 allows players to experience the end of the Human-Covenant war. It featured the most refined features of the series up to its release, including a map editing mode known as Forge. This game, released in 2007, marks the end of the original trilogy. Thanks to the release of numerous prequels afterwards, it was also the final time John-117 would star as Halo’s lead until Halo 4’s release in 2012. Halo: Spartan Assault - Year 2554 Taking place between Halo 3 and Halo 4, Spartan Assault is a top-down spinoff set during a time of peace. Ignoring the end of the war, Covenant forces led by Merg Vol battle against improved Spartan soldiers in a world-shattering conflict. Its lead, Sarah Palmer, is an active part of future titles, giving players some extra backstory as they play the Halo games in order. Halo 4 - Year 2557 Set four years after the events of the third game, Halo 4 continues the story of Master Chief as he battles against remnants of the Covenant. This title features new and improved multiplayer alongside updated weapons for fans to try out. It also includes a co-op mode known as Spartan Ops, continuing the story and giving the players even more great content to experience. Halo: Spartan Strike - Year 2552-2557 Spartan Strike is the eighth game in the chronological order of the Halo series. Its missions are set both during the events of Halo 2 and after the events of Halo 4. It follows the top-down gameplay of Spartan Assault while tying more closely into the events of the main series. Despite this, Spartan Strike doesn’t influence the story of the main series all that much; an understandable development, seeing as how it was designed for the sake of mobile phones. Halo 5: Guardians - Year 2558 Halo 5 once again follows Master Chief as he battles against rogue Covenant forces. During the campaign, players can issue orders to fellow NPCs in order to provide an even more robust solo experience. It also included Warzone, a massive gamemode with support for up to 24 players. Halo 5: Guardians was the last mainline game for six years, marking the longest time fans had to wait for the return of Master Chief. Halo Wars 2 - Year 2559 Set just after the events of Halo 5, Halo Wars 2 follows the RTS gameplay style of its predecessor. Players continue the story of the crew from the first Halo Wars, fighting against yet more Covenant remnants in a desperate bid to defend all of humanity. The story of this title doesn’t get much reference in the latest Halo entry, but it’s a fine choice for fans of the series that also love RTS games. Halo Infinite - Year 2560 Halo Infinite is the final title in this list of all the Halo games in order. This game features a campaign with an open world setting, a first for the series since ODST. Between this and other features, such as the return of split-screen play and a cross-platform online mode, Infinite is perhaps the most definitive Halo experience out there. The Halo series is more popular than ever, and the games only cover a fraction of it. The franchise is populated with novels and movies, providing backstory while allowing a glimpse into the future of Halo. One can only wonder where the series will go from here - and just how much further Master Chief’s story can be taken. More in gaming: 1. The Wolfenstein Games in Chronological Order and Ranked by Sales | HackerNoon 2. 10 Best Xbox 360 Games of All Time Ranked by Sales | HackerNoon 3. PC Gaming Accessories and Peripherals: A Starter's Guide | HackerNoon

All the Halo Games in Chronological Order

Destiny is known for its butter-smooth gameplay, profound lore and immersive storytelling. However, it is less known for its vast range of guns and weapons available. Destiny 2 has hundreds of different weapons, grouped into four different categories based on how rare they are: that being, common, rare, legendary and exotic. Common is the least rare weapon type you can get in the game, while exotic is the rarest you can receive. The game’s exotic weapons are hard to find, and all have unique perks for players to enjoy. For example, the infamous Thunderlord shoots down thunderbolts on enemies upon impact. The guns also vary in how powerful and valuable they are. Anarchy, an exotic grenade launcher introduced in the Forsaken expansion, is considered one of the most powerful and best exotics. So in this Destiny Guide, we’ll explain how to get Anarchy in Destiny 2. The article contains the following content: What is Anarchy How to get Anarchy in Destiny 2 Why Should I get Anarchy? What is Anarchy? Anarchy is an exotic heavy grenade launcher introduced into the game in the Forsaken expansion. The weapon is the best grenade launcher in the game, having the unique perk of sticking grenades to walls and chaining lightning between them. Thus, in PVP and PVE, the weapon is a machine. Due to its special bonus, Anarchy has the highest DPS of any heavy weapon on bosses and is best suited to completing raids and high-end game content such as dungeons. If you remember the legendary Gjallarhorn introduced in the first Destiny, Anarchy is its replacement, quickly becoming the new meta of the game. How to Get Anarchy in Destiny 2 There are several ways you can get Anarchy in the game. Firstly, you can obtain the weapon through what is known as the monument of the lost light. This monument is found in the tower and can be used to purchase exotic weapons at a price. To buy Anarchy, you will need an exotic cipher obtained through various means such as through Xur agent of the nine or completing triumphs. Likewise, you will need 150,000 glimmer, 2 ascendant shards, and finally 240 spoils of conquest obtained from conducting raids such as deep stone crypt. Once you acquire all of these, you can purchase Anarchy. Secondly, you can get the weapon through a random exotic engram drop. You can obtain these through the season pass, the crucible and by completing vanguard strikes and nightfalls. The chances of Anarchy dropping by these means are rare, though. Thirdly, Anarchy is obtainable through completing raids and opening the final raid chest. The chances of the weapon dropping vary between 10 and 50%, depending on the times you have completed the specific raid. The more times the raid has been completed, the higher the chance the weapon drops. This method is more time consuming than the others but is worth doing if you have a reliable fireteam with skill. Lastly, you could have obtained Anarchy by completing the scourge of the past raid. This raid was introduced after the Shadowkeep expansion and took place within the last city. Unfortunately, though, the raid was made unavailable last year. Why Should I Get Anarchy? As highlighted, Anarchy is one of the best weapons in the game; its incredible high DPS against enemies and players make this gun a must-have. If you manage to obtain the weapon, it is worth using the firearm in an arc class build to take advantage of arc damage. Similarly, matching the gun with exotic armour such as Dunemarches for the Titan is also a good move. It is crucial to make good use of its chain lightning. Doing so makes this gun a formidable opponent in both PVE and PVP. The weapon will make it far easier to complete high-level content such as raids and will make you a contender in the crucible in PVP. Final Thoughts This feature has covered how to get Anarchy in Destiny 2; as highlighted, Anarchy is tricky to find and can be a real challenge to obtain. However, if you get the weapon, you can become one of the game’s very best. For more Destiny 2 related features and articles, subscribe to our newsletter in the footer below. Also, check out, Destiny 2's Witch Queen Lives Amongst Us How to get the Destiny 2 Upgrade Module 13 of the Best Video Game Trailers of All Time

How to Get Anarchy in Destiny 2 (Destiny Exotic Guide)

In early March 2021, we announced our $5.2M seed round with Accel for - our open-source data integration platform; 2 months later, we announced our $26M Series-A round led by Benchmark. Airbyte In this article, we want to tell you about what happened behind the scenes, including the deck we presented to the Benchmark team. We hope this will give you some insights about the fundraising process, even though our case might be atypical. If you like what we do, don’t hesitate to subscribe to our or star ! newsletter our GitHub project If you're mostly interested in the deck, here's a . Slideshare link to the Series-A deck The Timeline Some additional context: We actually signed the term sheet for our seed round with Accel in December 2020, but we just closed the round (after the due diligence and all the negotiations on the long form contract) in early February. It then took us about a month to prepare the , which happened on March 2, 2021. announcement for the seed The First Call Benchmark contacted us at the end of February, and we had our first call with them on March 10. Although our seed round wasn’t public information, they already knew about it, as most VCs did. Private news travels fast in the VC world. We witnessed that again during the Series-A, as we got contacted by a lot of VCs after we signed the term sheet, even though we kept it confidential. One hypothesis is that the information leaks during the We’ll come back to that later. vetting process. How did the first contact happen? Chetan Puttagunta and Peter Fenton both reached out to Michel and John, via different mutual connections. Having both of them try to reach out to us was a strong signal about how serious they were about talking with us. Leading up to the Partnership Meeting After the first call on March 10, we had several video calls over the next month. Each one ranging from an hour to several hours. These long form conversations allowed us to get to know Benchmark and, at the same time, allowed Benchmark to get to know us as founders as well as our plans for Airbyte. We knew that we were going to take on a board member with the Series A and it’s critical to have clarity before starting the 10+ year journey together. We felt that this discussion had to happen over several weeks. We presented to the Benchmark partnership on April 12. That’s when we presented the deck below. We sent it the day before so they could prepare their questions and so we could leverage that hour the best. After that, we had calls on April 13 and 14 to negotiate the term sheet, and we got it signed on April 15. During that time, we didn’t interact with any VCs other than Accel. We didn’t need to raise money. And Accel was a great sounding board and was putting our company first. Vetting Process While you can be sure the VC will contact people you have worked with in the past, you want to talk to companies have invested in as well. We try to get feedback from companies in various states of progress: hugely successful, successful but in a slow growth mode, and some that didn’t work out. The goal is to obtain a well-rounded view of the VC’s support and behavior in all scenarios. We made all these calls between the 7th and 14th of April. they It might be possible the information leak happens at this stage. After all, the CEOs that you’re talking to are well connected with other VCs. The Term Sheet A piece of advice here: Try to get a maximum amount of information about the terms of the round written in the term sheet. Once you have signed the term sheet, most of the negotiation you will do will be with the VC’s lawyers rather than the VC partner. So, if you get it written in the term sheet, it’s one less thing that is open to negotiation. And don’t accept a one-page term sheet. Try to get the full set of terms clearly written. This is the only way to know as entrepreneurs that there will not be an egregious term in the final documents. Benchmark clearly spelled out the terms of their financing in their term sheet, which made the closing process smooth and on schedule. Post-Term Sheet As founders, you have two jobs to do after the term sheet is signed: Negotiate the details of the long-form investment agreement with the VC’s lawyers. That’s when having great lawyers helps, as they can make sure to represent your best interests. Complete the remainder of the round. You might have pro-rata rights to abide by for your earlier investor, and you might want to involve some high-profile business angels who can bring a ton of value, too. In the end, we got about a dozen more investors in the round, including Shay Banon (co-founder & CEO of Elastic), Dev Ittycheria (CEO of MongoDB), Auren Hoffman (co-founder of LiveRamp and Safegraph), and SV Angel. Why We Decided to Raise Even Though We Didn’t Need the Money Entrepreneurs often think in terms of valuation and the equity they own in the company. But there is a third factor that is the most important of all: the which goes from 0 (bankruptcy) to 1 (IPO or huge exit). success factor, success factor * ownership * valuation = value you own We did YC in January-March 2020 because we thought it would have a significant impact on our success factor. We chose to raise with Accel (we didn’t need that money then as we still had a full year of runway) because we thought they were the ideal partner to bring us closer to the 1 for our success factor. This is what happened again with the Benchmark team. We were not convinced at all about raising a Series-A now at the start, but each interaction with Chetan and the Benchmark team was bringing immense value and extremely relevant advice. After the third call, we could no longer see Airbyte without them working closely with us to bring us closer to the 1. Now that we knew whom we wanted to partner with, it was all about getting terms (valuation, etc.) that worked for both us and Benchmark. And that would be our advice in fundraising: Optimize first for whom you want to partner with. And then, optimize the terms of the fundraising (valuation included) for that partner. Our Deck As the was public, we couldn’t just reuse it. And a lot of progress had been made since we signed the seed term sheet in December, and we wanted to reflect that. Airbyte seed deck So here’s the Series-A deck we presented to the Benchmark partnership team: https://www.slideshare.net/jeanlaf/airbyte-seriesa-deck Our deck was pretty standard, though some might consider it a bit short. That’s why we include the appendices along with the deck. The Structure and Our Pitch on Each Slide Here are more details on our slides - why we included them and what we were saying during our pitches. 0. Cover Although you might only spend 10 seconds discussing the cover slide, it is important. It’s about showing that you have identified your positioning. In our case, we identify EL(T) as . So, essentially, we want to identify ourselves as the future of the ETL industry here. the future of the ETL industry 1. Industry Context This is the only slide that is common to our seed deck. Thanks to Snowflake’s IPO and Segment’s acquisition, we knew that data infrastructure was hot from an investor point of view. We decided to start with some context on the industry and how we fit into this ecosystem. What we were saying: When you look at the data infrastructure industry, there is often a new category that emerges through a commercial product. Once the market matures, an open-source alternative gets created and ends up taking over the category. This behavior is often seen because data infrastructure requires privacy, security and scale - which cloud-based solutions can’t offer as well as open-sourced ones. There are many examples, such as Kafka, Spark, and now dbt. We want to be the open-source solution for data integration. You might wonder why an open-source approach would also win the format for data integration; sometimes a closed-source cloud-based approach works. This last sentence is a transition to the next slide. 2. Problem We’re Solving This is a rephrasing of our seed round slide with more details. What we were saying: In June and July 2020, we started reaching out to 250 of Fivetran’s, StitchData’s and Matillion’s customers. We ultimately managed to talk to 45 of them. We wanted to know whether an open-source approach would make sense to address data integration. What we learned is that a cloud-based closed-source solution will never be able to fully address the data integration problem. It has several inherent issues. 100% of the companies we talked to were using Fivetran, StitchData or other solutions, while also building and maintaining their own connectors. They did so because either (a) the ETL solution didn’t support the connector they wanted, or (b) the solution supported it, but not in the way they needed. When you look at Fivetran, for instance, you’ll see that after 8 years, they only support 150 connectors. The hard part about ETL/ELT is not about building the connectors, but maintaining them. It is costly, and any cloud-based closed-source solution will be restricted by a ROI (return on investment) consideration. It isn’t profitable for them to support the long tail of connectors, so they only focus on the most popular integrations. During those 45 interactions, we also identified a third issue. Some of the companies were about to stop using Fivetran for some connectors because it was becoming too pricey. The value of an ELT solution is about replacing a paid data engineer who builds and maintains a connector in-house. The amount of work required from an engineer is about the same, whether the volume of data being moved is low or high. So with volume-based pricing, at some point it just stops making sense to use an external solution. And the last inherent issue with a cloud-based approach: Although cloud data warehouses are winning the enterprise market, it is because they are considered part of the data infrastructure. All other solutions must go through a rigorous privacy compliance process that will take several months. 3. Our Solution That’s the slide where we start to get deeper into our differences from our seed deck. What we were saying: There are two main differences with our approach. 1. Being open-source: Enables us to address the long tail of integrations: We built a (CDK) that enables our users to build new connectors in a matter of hours (instead of days!) for our community. The idea is to standardize the way connectors are being built, thus making them much easier to maintain by us and the community. Connector Development Kit Enables our users to customize any connector to their specific needs, and to leverage Airbyte the way they need to, along with other tools in their data stack. Enables our users to start testing Airbyte without asking upper management for permission. So the speed to value in a testing use case is way better than any closed-source solution. Enables us to address the use case of all companies that can’t use a third-party vendor to handle their sensitive data. This includes healthcare, fintech and insurance, as well as enterprises. 2. Current solutions have adopted a -based pricing, which has always been counterproductive for data. Having a value-based pricing. volume We won’t detail much yet what our intentions are here, as this is something that we would like to keep under the hood. 4. Our Product What we were saying: Airbyte addresses two different audiences. With our , we want to empower , such as data analysts and scientists, to start replicating data in a no-code way. Airbyte essentially makes them autonomous; they are no longer dependent on the priorities of the data or software engineering team. It takes literally for a data analyst to replicate data from Salesforce to Snowflake, including the deployment through our Docker Compose. easy-to-use UI data consumers 2 minutes 5. Our Product [2] What we were saying: The second audience we’re addressing is With our and soon our , we enable them to leverage Airbyte to power the data replication they need in their software and workflow. Some will even leverage Airbyte to offer data integrations on their own platform to their customers. data and software engineers. API CLI Airbyte integrates with dbt, Airflow and Kubernetes for now. But our intention is to integrate with the whole stack by the end of the year. Our goal is that Airbyte just works - whatever your data volume, architecture, infrastructure, or connector needs - by the end of the year. 6. Our Team In this slide, we review the expertise of the team. In our case, we have six people coming from Liveramp, where they built and maintained more than 1,000 connectors and were moving more than 100TB of data every day. Our goal is to make investors understand that we’re experienced and have already done it in the past, but now the goal is to enable company to do it. every Since we presented these slides we’ve hired three more people: Natalie K. (Director of Operations), Jenny B. (Senior Software Engineer) and George C. (Senior Software Engineer). By the way, we are hiring! 7. Our Velocity This is possibly the most important slide, as it validates everything we’ve said with facts. Please keep in mind that these metrics are the ones we had on April 11, when we presented the deck to the Benchmark partnership team. The point we wanted to make is how far we’ve come since the seed round’s term sheet was signed. We deliberately didn’t choose the seed closing date, because Accel made the decision to invest in Airbyte in December, not in February. The number of companies using you is most likely the most important data point for investors. What we were saying: We actually started working on Airbyte at the end of July. We soft launched a MVP 2 months later (at the end of September), with only 6 connectors and no support for incremental replications. We wanted to have feedback as early as possible. Since then, we’ve been used by X companies to sync data, we have Y PR contributors and Z issue contributors, and we’ve ramped up the number of connectors to 60. 8. Some Growth Metrics Investors will always ask for the names of companies that are using you. So here, we’re just anticipating the question. Our point on this slide is to show that the conversion rate from users having synced data to users who are using us in prod every day is growing every week! 9. Our Roadmap Now that we’ve shown our velocity up to this point, it’s time to show what we can accomplish in the next few quarters. Investors want to see where you will be when it is time to raise the next round. That is this slide’s goal. 10. Our Ask This is the last slide of the pitch. The goal here is that you shouldn’t need to say much! It shouldn’t come as a surprise, and is a direct consequence of the roadmap slide. 11. Appendix! You need some sort of slide to separate the main deck from the appendix. No surprise here! We only showed the appendix slides if asked relevant questions, but we sent the deck with the appendix. 12. The Competitive Landscape Here are a few choices we made that distinguish Airbyte from other open-source solutions: What we were saying: So the velocity in terms of execution, team and community growth are not comparable. Airbyte’s connectors are usable out of the box through a UI and an API, with monitoring, scheduling and orchestration. It takes literally 2 minutes to start replicating data from Salesforce to Snowflake. Airbyte runs connectors as Docker containers, so they can be built in the language of your choice. Airbyte’s components are modular, and you can decide to use subsets of the features to better fit in with your data infrastructure (e.g., orchestration with Airflow or Kubernetes or Airbyte…) We integrate with dbt for the transformation piece and will soon let the community contribute normalization schemas for all connectors. Unlike Singer, Airbyte uses one single open-source repo to standardize and consolidate all developments from the community, leading to high quality connectors. We built a compatibility layer with Singer so that the Singer community can easily migrate their work to Airbyte. The point for us here was to demonstrate that we are well on our way to becoming the industry’s open-source standard. 13. Some Growth Metrics [1] Even though GitHub stars are essentially vanity metrics, they are an indicator of the awareness you create. And it adds more support in demonstrating growth to your investors. 14. Some Growth Metrics [2] These metrics are much more important to show, as our defensibility will come through the community. The exponential growth was what was exciting to investors. 15. Some Growth Metrics [3] We anticipated that Series-A investors might need more metrics than we included in the seed deck. We showed the growing usage per user. If we make our prod users successful, they will naturally put more and more of their connectors on our platform when moving away from other solutions. 16. Some Growth Metrics [4] Even though we never actually presented this slide, it was important for us to include it in the deck. As we sent the deck before the meeting, it was already answering the question, “Who is using you in prod?” 17. Our Investors This is probably the slide we showed the least often, as serious VCs will already know who your main investors are. But you should still have it ready just in case. That’s it, folks! Now you know why we included the slides we did, and what we were trying to accomplish with each one. Hope this helps you understand us better, and maybe helps you prepare for your own investor encounters. Let us know what you think or if you have any questions in the comments below! Previously published at https://airbyte.io/blog/how-airbyte-raised-its-series-a-round-2-months-after-its-seed

Valuable Insights About The Fundraising Process

When you’re selling or considering purchasing a B2B tool, you need to understand the build vs. buy argument. What are the pros and cons of building the tool internally vs. buying the tool from a third-party vendor? This is especially true in big companies where you have the resources to build the said tools. Early-stage startups will generally opt for the faster route, going with self-served B2B tools -- unless the pricing is prohibitive. But something we don’t often think about is how open-source just messes the whole thing up. The build is completely redefined. You now need to compare the B2B tool with the build without the open-source tool, as well as with the open-source tool, which most often lowers the barrier significantly. In this article, we’ll take the example of the ETL/ELT industry. We know it best, as we’re building Airbyte, the open-source ELT alternative. Let’s see how open-source for ETL / ELT with Airbyte is also flipping the previous Build vs. Buy balance on its head. We’ve produced an infographic to illustrate that point. You will see that without taking Airbyte into consideration, the build vs. buy was pretty useful with Fivetran, in contrast to building connectors yourself. But now, with Airbyte, you can either just use the open-sourced connectors and start replicating data in minutes for free, or even build new connectors (if ever Airbyte doesn’t support them) in a matter of days (vs. months before) with maintenance being crowdsourced throughout the Airbyte community. The Infographic Here is: in white, the original “build” scenario; in blue, the original "buy" scenario with cloud-based Fivetran; in purple, the new "build" scenario with 2 options: “build non-supported connector with Airbyte” in light purple, and “use prebuilt connectors from Airbyte” in dark purple Let’s just say it: the playing field has changed! The Explanation Some context: the average business today uses well over , many of which contain valuable insights about an organization’s operations. Your company is likely on the way to using just as many apps, if not more, and you’ll need a solution to integrate all of the data your apps produce. 100 software apps Time & Effort Building your own pipeline by yourself is a significant time commitment. It can take between 3-6 months to set up a basic pipeline. Furthermore, beyond the time commitment, there is some inherent complexity in building a reliable, high-performance ELT pipeline. You need to: Obtain developer access to the data source Explore the data Design the schema/data models Set up a connector framework Test the connector and validate the data Set up orchestration, configuration validation, state management, normalization, schema migration, monitoring, etc. that happens every few weeks. This part is very cumbersome, as it requires an increasing number of data engineers to manage your connectors. Maintain the connector for every schema change In contrast, an off-the-shelf solution such as Fivetran can be set up in a matter of minutes with prebuilt connectors. Airbyte also takes literally 30 seconds to deploy, and you can start replicating data . within 2 minutes The big difference between both options in terms of time and effort is that all the Fivetran customers we talked to also had to build and maintain connectors on the side, as the connectors they needed were either not supported in the way they needed or not supported at all by Fivetran. That’s where the option to build with Airbyte comes in. For connectors not supported by Airbyte, it is a matter of hours to build connectors. Indeed, Airbyte already took care of having a UI, monitoring, scheduling, orchestration, integration with your data stack, automatic schema changes, etc. There is a very high chance we support your destination. So in the end, it’s only the EL part of the source connector you have to build, and Airbyte is providing some abstractions to make that easier. Regarding maintenance, the goal of Airbyte is to crowdsource throughout the community. When a connector fails because of significant API changes, it will notify the connectors’ users. As soon as the fix is made available by the Airbyte team or a community member, Airbyte will propagate the fix to all the users. The hope is that this approach will provide a better SLA than closed-source solutions such as Fivetran, not to mention the fact that you won’t have to maintain the connector yourself. People & Money From what we’ve seen, a typical company requires the equivalent of at least two or three full-time data engineers to build and maintain a data pipeline. The total cost of three full-time engineers can reach the high six figures (including benefits). So that’s a lot! Fivetran’s fees for a typical mid-sized company with five connectors is about $50,000. But you’ll have to add to that cost all the connectors you need to build and maintain by yourself. In contrast, Airbyte’s connectors are open-sourced, so you can use them for free. You also don’t need to pay for the egress to Fivetran’s infrastructure. It is possible that you might need a little bit of engineering time to operate Airbyte. If you need to build some of the connectors yourself, you will have to pay for the time spent by the data engineering team on building and maintaining them, but that would still be way less than if you had to do everything yourself. Opportunity Costs The actual value brought by your data team is through analysis and modeling. All the data integration, cleaning and transformation is important, as they enable the analysis and modeling. So the more time your team can spend on value-producing tasks, the better for the business. So opportunity costs as depicted in the illustration are very important to consider. Plus, ask any data team -- they will much prefer doing analysis or modeling tasks, rather than pipelining! So you will have better talent retention this way. Now you can see how open-source can flip the previous build vs. buy balance on its head. Before Airbyte, Fivetran was an easy sell. Now, it seems the contrary. Leveraging Airbyte’s open-source technology to build your own data infrastructure seems the obvious choice. There is one last thing to consider when choosing which direction to take: the future. Future Growth of Your Company As your company grows, you will add data sources to the pool. The complexity and effort of building and maintaining a data pipeline for a huge number of data sources can quickly escalate beyond your data engineering team’s ability to handle it. You might consider taking a chance on Fivetran’s ability to cover all or most of your connector needs, so that your team doesn’t need to build and maintain a continually increasing number of connectors (that would defeat the purpose). But, be mindful that Fivetran will always have a ROI consideration to maintaining connectors on the long tail; they won’t maintain connectors that don’t bring enough revenue to offset the maintenance costs. On the other hand, Airbyte will continue to grow the number of prebuilt community-maintained connectors, and can even take a large portion of the maintenance costs off your hands. When making a decision, consider how your company will evolve. And you can be sure that a great data infrastructure that grows with you will be a competitive advantage. And this way of thinking is something that you should have for any tool. Previously published at https://airbyte.io/articles/data-engineering-thoughts/how-open-source-can-disrupt-build-vs-buy-considerations/

Open-source Effect On Build Vs. Buy

There are quite a few decks already available online, and we found them all useful, each in different ways. That’s why we thought the deck we used for our seed round could be helpful to some companies, especially those that are open source or developer tools. For context: on that seed deck, we started working on our , , in the end of July, and raised our seed round with Accel in December, so only 5 months in. We’re assuming the deck was acceptable, as we raised the seed after only 13 days in the fundraising process. open-source data integration platform airbyte.io We will write a dedicated article about the process within a few days, so this article is really about the deck itself. Our Deck Here’s the Slideshare . link Our deck was pretty standard, though some might consider it a bit short. That’s why we always sent the appendixes along with the deck after our meetings. Here are more details on our slides – why we included them and what we were saying during our pitches. 0. Cover Although you might only spend 10 seconds discussing the cover slide, it is important. It’s about showing that you have identified your positioning. 1. Industry Context Thanks to Snowflake’s IPO and Segment’s acquisition, we knew that data infrastructure was hot from an investor point of view. We decided to start with some context on the industry and how we fit into this ecosystem. What we were saying: When you look at the data infrastructure industry, there is often a new category that emerges through a commercial product. Once the market matures, an open-source alternative gets created and ends up taking over the category. This behavior is often seen because data infrastructure requires privacy, security and scale, which cloud-based solutions can’t offer as well as open-sourced ones. There are many examples, such as Kafka, Spark, and now DBT. We want to be the open-source solution for data integration. You might wonder why an open-source approach would also win the format for data integration; sometimes a closed-source cloud-based approach works. This last sentence is a transition to the next slide. 2. Problem We’re Solving What we were saying: In June and July, we started reaching out to 250 of Fivetran’s, StitchData’s and Matillion’s customers. We ultimately managed to talk to 45 of them. We wanted to know whether an open-source approach would make sense to address data integration. What we learned is that a cloud-based closed-source solution will never be able to fully address the data integration problem. It has several inherent issues. 100% of the companies we talked to were using Fivetran, StitchData or other solutions, while also building and maintaining their own connectors. They did so because either (a) the ETL solution didn’t support the connector they wanted, or (b) the solution supported it, but not in the way they needed. When you look at Fivetran, for instance, you’ll see that after 8 years, they only support 150 connectors. The hard part about ETL/ELT is not about building the connectors, but maintaining them. It is costly, and any cloud-based closed-source solution will be restricted by a ROI (return on investment) consideration. It isn’t profitable for them to support the long tail of connectors, so they only focus on the most popular integrations. During those 45 interactions, we also identified a third issue. Some of the companies were about to stop using Fivetran for some connectors because it started to become too pricey. The value of an ELT solution is about replacing a paid data engineer that builds and maintains a connector in-house. The amount of work required from an engineer is almost the same whether a low volume or a high volume of data is being moved. So with volume-based pricing, at some point it just stops making sense to use an external solution. And the last inherent issue with a cloud-based approach: although cloud data warehouses are winning the enterprise market, it is because they are considered part of the data infrastructure. All other solutions must go through a rigorous privacy compliance process that will take several months. 3. Our Solution What we were saying: We’re building an open-source ELT platform that syncs data from SaaS apps, APIs and databases to data warehouses, data lakes and other databases. Our solution can fully integrate with your data infrastructure and stack, if you are using Kubernetes or Airflow for orchestration, or DBT for transformation. Our goal is to become the open-source standard for anything ELT by the end of 2021. 4. How it addresses the problem What we were saying: By making it trivial to build and maintain connectors using Airbyte rather than doing it in-house, we will become the new standard for building connectors. This will help us support the long tail of connectors. And since connectors run as Docker containers, you can build them in the language of your choice. As connectors are open-sourced, any team can edit a pre-built connector and tune it to their needs. If a connector breaks, anyone can jump on the code, submit a PR and, once approved, the change can be propagated across all the existing users of that connector. Airbyte focuses first on a self-hosted offer;  pricing will be based on the feature and number of connectors used. The pricing will not be indexed on the volume of data. Being open sourced enables us to have a bottom-up approach with a frictionless adoption from data teams, without going through privacy compliance, as we handle data security as a first-class citizen. 5. Who We’re Targeting / Our Go-To-Market What we were saying: With Airbyte, we wanted to address 2 different audiences: Data consumers, including data analysts and scientists.Data engineers who were building and maintaining the connectors themselves, or managing the data infrastructure. What made Fivetran successful is that they enabled data consumers to leverage the data without the help of data engineers – they made them autonomous (as much as Segment made product teams autonomous). Airbyte provides a UI that makes it very easy to start replicating data for non-technical users. It takes literally for a data analyst to replicate data from Salesforce to Snowflake, including the deployment through our Docker Compose. 2 minutes In addition, Airbyte will offer data integration connectors through an API that data engineers can leverage to build their own workflow and applications. This is also a way for Airbyte to address SaaS businesses that want to offer their own integrations to their customers through Airbyte. 6. Our Team In this slide, we review the expertise of the team. In our case, we have 4 people coming from Liveramp, where they built and maintained more than 1,000 connectors and were moving more than 100TB of data every day. Our goal is to make investors understand that we’ve already done it in the past, but now the goal is to enable every company to do it. 7. Our Velocity This is possibly the most important slide, as it validates everything you’ve said before with facts. The number of companies using you is most likely the most important data point for investors. What we were saying: We actually started working on Airbyte in the end of July. We soft launched a MVP 2 months later (at the end of September), with only 6 connectors. We wanted to have feedback as early as possible. It’s been 6-7 weeks since we soft launched, and we are now used by X companies, we have Y contributors, and we’ve ramped up the number of connectors to 43. 8. Our Roadmap Now that we’ve shown our velocity up to this point, it’s time to show what we can accomplish in the next few quarters. Investors want to see where you will be when it is time to raise the next round. This slide’s goal is to accomplish this. 9. Our Ask This is the last slide of the pitch. The goal here is that you shouldn’t need to say much! It shouldn’t come as a surprise, and is a direct consequence of the roadmap slide. 10. Appendix! You need such a slide to separate the main deck from the appendix. No surprise here! We only showed the appendix slides if asked relevant questions. 11. Some growth metrics [1] The point for us here was to insist that we hadn’t done a hard launch yet, only a soft launch on our social networks on 09/24. The rest of the growth is organic. 12. Some growth metrics [2] Investors will always ask for the names of companies that are using you. So you should always have such a slide ready. The more recognizable the names of the companies are, the better. 13. The competitive landscape What we were saying: Here are a few choices we made that distinguish Airbyte from other open-source solutions: Airbyte’s connectors are usable out of the box through a UI and an API, with monitoring, scheduling and orchestration. Airbyte runs connectors as Docker containers, so they can be built in the language of your choice.Airbyte’s components are modular, and you can decide to use subsets of the features to better fit in with your data infrastructure (e.g., orchestration with Airflow or K8s or Airbyte’s…)We intend to integrate with DBT for the transformation piece soon and let the community contribute normalization schemas for all connectors. Unlike Singer, Airbyte uses one single open-source repo to standardize and consolidate all developments from the community, leading to higher quality connectors. We built a compatibility layer with Singer so that Singer taps can run within Airbyte. 14. Our investors This is probably the slide we showed the least often, as serious VCs will already know who your main investors are. But you should still have it ready just in case. That’s it, folks! Also published here .

All 14 Slides from the Deck We Used to Raise Seed with Accel in 13 Days

A lot of good movies are coming out this year, and the new Mortal Kombat movie still manages to be one of the films I’m most excited about. As a huge fan of the franchise, I can’t wait to see these legendary characters brought to life. With the trailer showing them off, my hype is at an all-time high. But I hope more characters make an appearance than the ones shown in the trailer. They don’t need big roles, but some cameos or some mentions would be awesome. Here are the 5 characters who should make an appearance in the new Mortal Kombat movie in 2021. 1. Baraka This Tarkataan warrior has never made an appearance in a live-action Mortal Kombat movie, and I think it’s time Baraka got his time to shine. His razor-sharp teeth and his arm blades have made him one of the most unique characters in the franchise. He’s mostly always sided with the bad guys, and it would be cool to see our heroes try to take him down. Baraka’s had rivalries with Kung Lao and Mileena in the past, and it would be interesting to see if they ever revisit that in a film. 2. Fujin Everyone knows Raiden, the god of thunder, but not everyone knows of Fujin. As the god of wind and Raiden’s brother, Fujin is a powerful fighter in his own right. He’s there to back up his brother as they fight the forces of evil. Although he first appeared in Mortal Kombat 4, Fujin hasn’t been a prominent character in the games until recently. Fans were clamoring for him to be a part of Mortal Kombat 11, and they got their wish. Now, my wish is to see Fujin in live-action, and hopefully, that one will come true as well. 3. Smoke With the inclusion of Sub-Zero in the movie, I hope his Lin Kuei clan also plays a big role in the film. Especially since one of the members, Smoke, is a fan favorite. His power is, you guessed it, turning into smoke. Friends with the younger Sub-Zero, Smoke was originally captured and turned into a robot. That was until some time travel shenanigans happened, and he managed to avoid that fate. Only to be killed and turned into a dead slave. Whichever story they decide to tell, Smoke and Sub-Zero’s friendship is something we have to see on the big screen. 4. Kenshi My favorite Mortal Kombat character, and in my opinion the coolest, Kenshi is a blind telekinetic swordsman. If that description doesn’t make you fall in love with this character, nothing will. Losing his vision after being tricked by Shang Tsung, Kenshi vowed revenge on the sorcerer. We already know Shang Tsung is in the movie, so it could be real easy to give Kenshi a quick cameo or a mention. Maybe there’s already too many heroes in the film to include him, but they could at least put down the seeds for his eventual appearance. 5. Johnny Cage Nobody should be surprised by this one. He is one of the original six characters of the arcade game, and someone who’s been in almost every installment; Johnny Cage needs to be in the film. He starts off as an arrogant Hollywood actor who becomes one of Earthrealm’s greatest heroes. It doesn’t need to be a speaking role; he doesn’t even need to be seen. I would be happy with just a mention of one of his movies, or even just a poster in the background. Something to let us know that he exists in this universe and that we can expect him in the future. Mortal Kombat has a deep roster of incredible characters, and I would love to see any one of them in live-action. However, these are the ones that I believe should make an appearance in the new Mortal Kombat movie. More News in Gaming: 1. Aliens: Fireteam Arrives This Summer with Third-Person Co-op Shooter Gameplay 2. Is Virtual Reality Still A Niche Within the Games Industry? 3. Fallout 4 Project Valkyrie: Depravity and the Thuggyverse

The New Mortal Kombat Movie in 2021: 5 Characters We Want to See

Open-source data integration is not new. It started 16 years ago with Talend. But since then, the whole industry has changed. The likes of Snowflake, BigQuery, and Redshift have changed how data is being hosted, managed, and accessed, while making it easier and a lot cheaper. But the data integration industry has evolved as well. On one hand, new open-source projects emerged, such as in 2017. This enabled  more data integration connectors to become accessible to more teams, even though it still required a significant amount of manual work. Singer.io On the other hand, data integration was made accessible to more teams (analysts, scientists, business intelligence teams). Indeed, companies like benefited from Snowflake’s rise, empowering non-engineering teams to set up and manage their data integration connectors by themselves, so they can use and work on the data in an autonomous way. Fivetran But even with this progress, a large majority of teams still build their own connectors in-house. The build vs. buy leans strongly on the build. That’s why we think it’s time to have a fresh new look at the landscape of the open-source technologies around data integration. However, the idea for this article came from an last week. The discussion centered around two things: awesome debate on DBT’s Slack The state of open-source alternatives to Fivetran, and Whether an open-source (OSS) approach is more relevant than a commercial software approach in addressing the data integration problem. Even Fivetran’s CEO was involved in the debate. We already synthetized the second point . In this article, we want to analyze the first point: the landscape of open-source data integration technologies. in a previous article TL;DR Here is a table summarizing our analysis. In orange is what we’re currently building at Airbyte in the next few weeks. To better understand this table, we invite you to read below the details of our analysis on the landscape. Data integration open-source projects Singer was launched in 2017, and was until now the most popular open-source project. It was initiated by StitchData, which was founded in 2016. Over the years, Singer grew to support 96 taps and targets. Singer : Talend (acquirer of StitchData) seems to have stopped investing in maintaining Singer’s community and connectors. As most connectors see schema changes several times a year, more and more Singer’s taps and targets are not actively maintained and are becoming outdated. Increasingly outdated connectors : each connector is its own open-source project. So you never know the quality of a tap or target until you have actually used it. There is no guarantee whatsoever about what you’ll get. Absence of standardization : you still need to build everything around to make them work. Singer’s connectors are standalone binaries , as some connectors are only offered by StitchData under a paid plan. No full commitment to open sourcing all connectors In the end, a lot of teams will use StitchData for the connectors that work well, and will build their own integration connectors if they don’t work out of the box. Editing a Singer connector is not easier than building and maintaining the connector yourself. This defeats the purpose of open source. Airbyte was born in July 2020, so it is still new. It was born out of frustration with Singer and other open-source projects. It was built by a team of data integration veterans from Liveramp, who individually built and maintained more than 1,000 integrations, so 8 times more than Singer. Their ambition is to support 50+ connectors by the end of 2020, so in just 5 months since the inception of the project. Airbyte Airbyte’s mission is to commoditize data integration, and we have made several significant choices towards this goal: . Airbyte was built on the premise that a user, whatever their background, should be able to move data in 2 minutes. Data engineers might want to use raw data and their own transformation processes, or to use Airbyte’s API to include data integration in their workflows. On the other hand, analysts and data scientists might want to use normalized consolidated data in their database or data warehouses. Airbyte supports all these use cases. Airbyte’s connectors are usable out of the box through a UI and API, with monitoring, scheduling and orchestration : This will help consolidate the developments behind one single project, some standardization and specific data protocol that can benefit all teams and specific cases. One platform, one project with standards , as Airbyte runs them as Docker containers. Connectors can be built in the language of your choice to let teams use whatever part of Airbyte they want based on their needs and their existing stack (orchestration with Airflow, Kubernetes, or Airbyte, transformation with or again Airbyte, etc.). Teams can use Airbyte’s orchestrator or not, their normalization or not; everything becomes possible. Decoupling of the whole platform DBT with the promise not to hide some connectors behind paid walls. A full commitment to the open-source MIT project The number of connectors supported by Airbyte and its community is growing fast. Their team anticipates that it will outgrow Singer’s by early 2021. Note that Airbyte’s data protocol is compatible with Singer’s. So it is easy to migrate a Singer tap onto Airbyte, too. PipelineWise is an open-source project by Transferwise that was built with the primary goal to serve their own needs. They support 21 connectors, and add new ones based on the needs of the mother company. There is no business model attached to the project, and no apparent interest from the company in growing the community. PipelineWise : PipelineWise aims to reproduce the data from the source to an Analytics-Data-Store in as close to the original format as possible. Some minor load time transformations are supported, but complex mapping and joins have to be done in the Analytics-Data-Store to extract meaning. As close to the original format as possible : When source data changes, PipelineWise detects the change and alters the schema in your Analytics-Data-Store automatically. Managed Schema Changes : Data pipelines are defined as YAML files, ensuring that the entire configuration is kept under version control. YAML based configuration : No daemons or database setup are required. Lightweight : PipelineWise is using Singer.io compatible taps and target connectors. New connectors can be added to PipelineWise with relatively small effort. Compatible with Singer’s data protocols Meltano is an orchestrator dedicated to data integration, built by Gitlab on top of Singer’s taps and targets. Since 2019, they have been iterating on several approaches. They now have one maintainer for this project that is CLI-first. After one year, they now support 19 connectors. Meltano : Meltano has the same limitations as Singer’s in regards to its data protocol. Built on top of Singer’s taps and targets : Meltano was primarily built with a command line interface in mind. In that sense, they seem to target engineers with a preference for that interface. CLI-first approach : Meltano has recently built a new UI to try to appeal to a larger audience. A new UI : Meltano offers some deep integration with , and therefore lets data engineering teams handle transformation any way they want. Integration with DBT for transformation DBT : You can either use Meltano alone for orchestration or with Airflow; Meltano works both ways. Integration with Airflow for orchestration Related noteworthy open-source projects Here are some other open-source projects that you might have heard of, as they’re often used by data engineering teams. We thought they deserved to be mentioned. Apache Airflow We see a lot of teams building their own data integration connectors using for the orchestration and scheduling. Airflow wasn’t built with data integration in mind. But a lot of teams use it to build workflows. Airbyte is the only open-source project to offer an API so teams can include data integration jobs in their workflows. Airflow DBT is the most widely used data transformation open-source project. You need to be proficient in SQL to use it properly, but a lot of data engineering / integration teams use it to normalize raw data coming into the warehouses or databases. DBT Both Airbyte and Meltano are compatible with DBT. Airbyte will offer teams the ability to choose between raw or normalized data for each connection they need, which addresses the needs of both data engineering and data analyst teams. Meltano doesn’t provide normalized schemas, and relies solely on DBT for that. Apache Camel is an open-source rule-based routing and mediation engine. That means you get smart completion of routing rules in your IDE whether in your Java, Scala, or XML editor. It uses URIs to enable easier integration with all kinds of transport and messaging models including HTTP, ActiveMQ, JMS, JBI, SCA, MINA and CXF, together with working with pluggable Data Format options. Apache Camel Streamsets is a data-ops platform that includes a low-level open-source data collection engine named DataCollector. This open-source project is not supported by any community, and is mostly used by the company to assure their enterprise clients that they will still have access to the code whatever happens. Streamsets Let us know if we missed any open-source projects or any valuable information on the listed ones. We will try to keep this list up to date and precise. Also published on: https://airbyte.io/articles/data-engineering-thoughts/the-state-of-open-source-data-integration-and-etl/

Open-Source Data Integration and ETL in 2020

By now you must have read quite a few articles about Snowflake’s absolutely mind-blowing and record-setting IPO. This article is not intended to speculate on whether the valuation makes sense or not, but rather help you understand the technological concepts that make Snowflake so unique, and why it has proven to be so disruptful for the data space in general and the data warehousing space in particular. Background: data warehouses start their journey to the cloud To understand the market Snowflake operates in, it’s important to first spend a few minutes on the history of data warehouses in the cloud. Data warehouses have been around forever on-premises, but their appearance in the cloud is still relatively recent, with the release of Amazon Redshift in 2013. It was the first cloud data warehouse that seemed to be headed towards becoming mainstream. It was only natural that AWS, as the leader of the cloud revolution, would release one of the first successful pure cloud data warehouses. Redshift quickly gained popularity and became the warehouse of choice both for companies that were transitioning from on-prem to the cloud (the name “Redshift” was a poke at Oracle — shifting from “Red” which is Oracle’s color), and for new cloud-first companies. The biggest benefit that ca9`me with Redshift was the fact that it is a managed cloud service. This meant that the ease of use of the cloud and SaaS finally became available to data warehousing. Provisioning a new warehouse suddenly was not a lengthy and complex project involving hardware and software, but rather a few mouse clicks with Redshift taking care of the heavy lifting behind the scenes. But beyond reducing complexities around installations, provisioning and maintenance, Redshift does the same things in the same manner that traditional on-premises warehouses do. It also shares the same limitations. Google and Microsoft released cloud data warehouse offerings of their own (Google’s BigQuery actually preceded Redshift), but neither was as successful as Redshift in grabbing market share and becoming the warehouse choice for on-prem to cloud transitions. ‘Decoupled storage and compute’ and simplicity as the foundation for Snowflake’s popularity In the last few years you’ve heard the term ‘elasticity’ a million times in the context of the cloud revolution. But the story of elasticity made a significant leap specifically for data warehousing thanks to the ‘decoupled storage and compute’ concept. Although there are many product capabilities behind Snowflake’s success, we will focus on decoupled storage and compute because it’s the most important of them all. Quick reminder — what is elasticity? A simple explanation of elasticity in the cloud can be found in : “the degree to which a system is able to adapt to workload changes by provisioning and de-provisioning resources in an autonomic manner…”. However, the fact that something is in the cloud doesn’t mean it automatically enjoys elasticity. This is especially true for data warehouses. Wikipedia What is ‘decoupled storage and compute’ and what does it have to do with elasticity? Decoupling storage and compute enables users to quickly change the type and size of compute (clusters of CPUs) that is assigned to be the processing unit over the storage layer (hard drives, SSDs, etc). The cloud made this approach easily accessible to software, but it took time for software products to leverage this architectural concept. Snowflake was the first cloud data warehouse that made it super simple for users to enjoy the benefits of decoupled storage and compute. No matter how much data you import into Snowflake, at any point in time you can decide which compute resource you want to use for your next workload with a few mouse clicks. For example — you might decide to use a small (and therefore relatively cheap) compute cluster to conduct some research over a sample of your data. Then you might decide to use a bigger (and therefore more expensive) cluster for production dashboards that are viewed by many people throughout the day. Since you’re typically billed per usage in the cloud, this approach provides granular control over your spend. You can literally choose on a per-query level which hardware to use. This is completely different from the traditional on-premise world, where the same hardware configuration you implemented for your warehouse always runs all queries, no matter what. Now back to elasticity — Snowflake’s approach to decoupled storage and compute, and making it easily accessible for users through no more than a few clicks, brought a new level of elasticity to data warehouses. From a product perspective, it is the key reason for Snowflake’s popularity among data professionals. What’s next for data warehouses in the cloud The benefit of decoupled storage and compute for data warehouses and cloud data technologies in general is undeniable. This is one of the key reasons users like Snowflake. Decoupled storage and compute . Amazon Redshift recently added both Amazon Spectrum and RA3 nodes for this reason. has become the architecture of choice for new data technologies As we built Firebolt on a decoupled storage and compute architecture, we also came to realize what was needed to change in storage and compute to solve many of the other problems with data warehouses, and there are plenty of them remaining. One of the bigger ones is being able to . Achieving fast query response times for big data sets is still something that modern warehouses struggle with, and when it’s possible, it typically comes with a price tag that only big enterprises can afford. Looking at public benchmarks comparing Snowflake, Redshift and BigQuery it’s clear that already in the 1TB scale of data, query runtime easily takes 7 seconds or more on average, sometimes even minutes ( ). As data keeps growing, and as being able to deliver great user experiences over data becomes increasingly important for businesses, ‘performance at scale’ limitations is what the market will be looking for the tech industry to solve. analyze huge data sets, quickly and easily without breaking the bank See Fivetran’s great benchmark post here A decade ago BI meant that a smaller group of analysts performed analytics against historical data using advanced BI tools to support decisions for the rest of the company. A decade from now we expect 100x or more employees and customers to be performing self-service analytics against historical and real-time data using newer tools that leverage machine and deep learning and other computing — all to help them make faster and better decisions on their own. Snowflake’s valuation reflects the future potential. But the data industry has a long way to go till it gets there. This is why from a technology perspective the future of cloud data technologies belongs to those that will be able to analyze more data at higher speeds, while relying on less hardware and compute power to do so, to reduce costs. This belief led us at Firebolt to start by concentrating on technology that can get much more analytic horsepower from existing hardware, as the foundation on top of which simplicity and the benefits of the cloud can bring powerful analytics to the masses.

Understanding the tech behind Snowflake’s IPO and what’s to come

There was an awesome debate on ’s Slack last week discussing mainly two things: DBT The state of open-source alternatives to Fivetran Whether an open-source (OSS) approach is more relevant than a commercial software approach in addressing the data integration problem. If you’re already on DBT’s Slack, here is . Even Fivetran’s CEO was involved in the debate. the thread’s URL In this article, we want to discuss the second point and go over the different points mentioned by each party. The first point will come in another article. It’s more relevant to discuss whether an OSS approach makes sense before drilling down into the different alternatives. We’ll go over the main challenges that companies face and see which approach fits best. We’ll call “commercial companies” the ones with a commercial software product, and “OSS companies” the ones with an open-source approach. 1. Having a large number of high-quality well-maintained pre-built connectors This might be the most challenging part for the open-source approach, but there are actually choices that can make an OSS approach even stronger than a commercial one on that matter. Commercial approach In this case, a company supports a limited number of connectors (the most used ones) and actively maintains them. They know when there is a schema change and when they need to update the connector, and can be pretty responsive on that, if the organization is responsive and scales well. However, the more connectors, the more difficult it is for a commercial company to keep the same level of maintenance across all connectors. In an ideal world, the organization will grow linearly with the number of connectors. But most often, there are inefficient processes, so every organization will reach a limit. The more efficient they are, the higher the limit. Open-source approach When you look at Singer.io, you would think that an OSS approach is at a serious disadvantage here. But Singer.io has actually made several significant poor choices here: There is an absence of standardization / enforcement of protocol, and developers just add whatever they want in their implementation and messages. Integrations live by themselves in their own repos. So each connector can be seen as an individual open-source project. This does not help when building a community. There is no real ownership, nor direction for the project. So can we actually achieve a large number of high-quality well-maintained connectors with an open-source approach? The answer is , but only if the following conditions are met: probably One single repo for the whole project and all connectors. This will help consolidate the developments, and it will unify the community behind one single project and one vision. This means that the project and community can be opinionated about what a connector should be, and how it should be built. Detailed logs for all users, with the ability to easily surface the issue on the GitHub project to the whole community. A separation between extract-load and transformation. Users would be able to use the connector without any normalization, so they use their own transformation processes. By doing so, we can more easily identify where failures happen, and how to solve them. Plus, it would make it easier to support a lot more integrations faster. A last note here: when users use a connector on prod and see it fail, they will be able to fix it without waiting for external support teams (as for commercial companies), and the whole community will be able to benefit from the fix. 2. Addressing unique corner cases and unique needs Every company is opinionated about the way it wants to use a connector; each has specific needs. Commercial approach Commercial companies have a ROI-driven relationship with connectors. Any additional work on any edge case will need to have some ROI, or the company will have a hard time scaling. That also means that the commercial company will most likely have very high quality connectors for the most used ones. But it can’t afford to keep that level of quality across all integrations, and thus won’t be addressing their corner cases. With their finite resources, it will always be a trade-off. Commercial companies can’t address the long tail of integrations and can’t meet all unique needs. Furthermore, in the case a client’s needs are not met, the client won’t be able to customize the existing connector to their needs. That is the limitation of this approach. Open-source approach It is actually not obvious that an OSS approach would easily address all unique corner cases. Contributors build the connectors with their use cases and needs in mind; they won’t be building a high quality connector to address most companies’ needs. However, there are ways to avoid that: Setting standards and enforce protocols, with one single platform / project, as mentioned in the first point. Decoupling extract-load (EL) and normalization. A lot of the customization can be done directly in normalization, but a lot of the EL is common to everyone. In that case, everybody can benefit from the EL part and use their own transformation tools to address their own needs. Enabling contributors to build their connectors using the language of their choice by running connectors as Docker containers Being open source lets any engineer adapt any pre-built connectors to their needs, so every company will be able to address their own needs. The question is more about how much effort would be needed for each of their cases. 3. Debugging autonomy This is an easy one. When dealing with a commercial product, if you face an issue, the only thing you can do is contact their support, and hope that your issue will be prioritized in the list of issues the company’s engineering team must resolve. You won’t be able to know when the issue will be fixed. It can take hours at best, or months. With open source, you don’t need to wait on anybody. You can fix the integration yourself. Having this ability is important if the integration is vital to your business. 4. Getting the internal resources By resources, we mean either budget or the engineering time to work on the integrations. At , we don’t like the word when talking about people. But, in this context. it is the only common one we could think of. Airbyte resources The truth is that in a lot of enterprises, it can be easier to justify hiring a new employee than getting even a modest budget approved for an external vendor. Depending on the open-source project, most or all of the job could be already done. That’s one of the reasons why most companies use open-source technologies rather than external vendors. 5. Easy to use by any teams (analysts, finance teams, etc.) Once consolidated, the data is used by different teams: business intelligence / analysts, data scientists, finance teams, customer success, product or marketing team, etc. That is the beauty of products like Fivetran: anybody can use them and anyone can add more data. One would think that ease of use is exclusive to commercial products, but that is actually not true. For instance, Gitlab and many other OSS projects have UIs that are usable by a non-technical audience. Airbyte has one and it is one of its primary features. 6. Security and privacy compliance We’re talking about data, and most probably this involves customer data. All companies are subject to privacy compliance laws, such as GDPR, CCPA, HIPAA, etc. As a matter of fact, above a certain stage (about 100 employees) in a company, all external products need to go through a security compliance process that can take several months. With open source, teams just use the technology directly without such processes. The adoption is frictionless, and the engineering need can be met overnight. 7. Moving data between internal databases Commercial products mostly sit in the cloud. So if you have to replicate data from an internal database to another, it makes no sense to have the data move through the external vendor. In addition to the vendor’s costs, you need to add egress costs and the issue it poses in terms of privacy. Open-source products are mostly self-hosted, so they don’t have this problem. Is open source hard without a direct revenue channel? One of the comments in the thread was that “open source is hard without a direct revenue channel to support it.” That is actually true for any company. If you have a hard time getting revenues, you can’t scale the team or your effort. According to Bessemer, in revenues than many cloud leaders. open-source companies grow faster Some other interesting quotes / points “The general argument for open-source software that applies to all software: Having access to the source code means users have freedom-the freedom to change the software how they see fit and fix issues.” “If you spent months or years using, say, Fivetran, and all of a sudden Fivetran goes out of business or significantly changes its business, you could run the software yourself, even if just for a period of time. I’ve been there when it’s happened with other software; it’s not pretty, and sometimes you only have days or weeks of heads up.” “If it’s open source, I can run it without Fivetran, so for special circumstances, like compliance or high volume, I don’t have to deal with legal or cost limitations.” “I can see the code running my integration; I can more quickly debug issues. I can also easily create integrations anyone can choose to use. Tools like Fivetran and Stitch only work a percentage of the time; open source allows me to fill in that gap without creating one-off scripts (or Fivetran lambda functions) or going back and forth for weeks or months with support on a bug that I could fix in a few hours.” Conclusion If the open-source project is well structured and well thought through, there will be no points where a commercial approach can be better. The issue is that, until now, there were no good open-source projects that could alleviate the dangers to this approach. If you like how we think an open-source project can address all the possible concerns by building a single platform with a single repo, a defined data protocol, a ready-to-use UI, and decoupling of EL and normalization, you should definitely have a look at what we’re building at . We want to become the open-source standard for data integrations. Airbyte Let us know if you think we missed anything. Our goal is to see things from all perspectives and to keep this article up to date. Also published at: https://airbyte.io/articles/data-engineering-thoughts/open-source-vs-commercial-software-how-to-better-solve-data-integration/

Stories, News & Mentions about FIVETEAMS

Fiveteams’s Stories on HackerNoon