Zipline is a Silicon Valley-based logistics company that designs, manufactures, and operates small robot aircraft to deliver essential medical products

Zipline

zipline

Everything is easily accessible when your home is in a city. Ever thought of people who live in small towns around the cities where they don't have access to basic healthcare services? According to the research of "Our world in data," around 15% of the population lives in rural isolated areas with fewer hospitals and lack of blood and vaccines. In the world of technology where people are still struggling for medicines & blood, Zipline, a Rwanda based company which is founded by Keller Rinaudo, William Hetzler, and Keenan Wyrobek has found a way to solve this problem. They have invented electric autonomous drones in the shape of an aircraft that can carry medical supplies to people in need, in less time. The Zipline aircraft was first invented in 2014 with the carrying capacity of 1.6 milligrams, but their latest drone aircraft version - introduced in 2019 -can carry up to 2 kg of medical supplies at a speed of 128 km per hour up to the range of 160 kms round trip. How it works The Zipline distribution center is 20kms outside ). Zipline also stores blood and medicines in its distribution center which is delivered to them by the national center of blood transfusion. They can deliver it to hospitals in just 20-30 mins. Kigali (the capital city of Rwanda When a hospital is in an emergent need of blood they send a message on WhatsApp to the Zipline team which then takes the blood from their storage and scans it into their systems so that the ministry of blood also has a record of where the blood has been transferred. The blood or medicine is packed in an insulated cardboard box with a small parachute attached to it. The Zipline aircraft is not always in a ready to launch state, All the parts of Zipline aircraft are detached an assembled on the launcher itself so that it becomes easy for the Zipline team to carry the parts individually. It has 4 parts that need to be assembled before the launch: Cargo compartment - The blood is first placed inside the cargo compartment. It is a kind of storage box where blood or medicine is kept Wings - After the order is kept inside the fuse lodge, wings of the aircraft are attached to the body of the aircraft. Battery - Once the wings are attached to the aircraft, then the charged battery pack is slotted into the aircraft. GPS - GPS circuitry is in the battery instead of the plane this way GPS is never off and always connected which saves GPS-satellite connection time. Once the assembling process is completed Zipline aircraft undergoes pre-flight checks which include checking every control surface with the help of a mobile app. Each and every control surface of aircraft has a QR code, Zipline technicians scan the QR code of every control surface and then this app runs the computer vision algorithm which results in a pass or fail judgment. If any control surface fails to pass the pre-flight check than only that control surface is dismantled from the aircraft and is changed. When the aircraft passes all the pre-flight checks it is launched using an electric catapult which produces a strong magnetic field to safely get the aircraft up to the speed. It accelerates the plane up to 100 km/hr in just 0.3 secs. When the aircraft reaches the destination it simply drops the order which with help of parachute comes to the ground and is safely received by the intended person. Aircraft then returns to the Zipline distribution center, where it is caught with a wire by snagging a 3cm long metal hook on the aircraft tale. Zipline drone landing is one of the most difficult tasks. Zipline has become one of the most successful tech-for-good companies in the world. President Kagame poses for a picture with the Zipline team and partners | Muhanga, 14 October 2016. Source: flickr According to drone.com, Zipline has done 14000 deliveries and saved the lives of millions of people by providing access to healthcare products in rural hospitals. Now Zipline is planning to expand its services to other countries in Africa, South Asia, Southeast Asia, and America. Zipline has shown people how technology if used effectively, can do wonders for every person on the planet. How might we all use tech for good in 2020 and beyond?

How Zipline Drones Can Save Lives in Rural Rwanda

The final frontier of delivery is going autonomous. As online orders surge and customers demand ever faster times, last-mile logistics is transforming at warp speed. Self-driving cars, drones and robots are invading suburbs worldwide, promising to bring goods to your door within minutes. No longer fiction, the vision of ordering coffee on a whim and seeing it arrive piping hot via robot is becoming reality. Need a package ASAP? Drones may soon deliver overhead. Companies like Yandex are accelerating this revolution by developing cutting-edge autonomous vehicles for last-mile tasks. In this article, we'll take a drive exploring the innovations upending last-mile delivery. We'll look at the technologies and companies at the forefront of this rapid change. We'll also peer into the future to speculate how new advances could push instant gratification to new heights. From Horse-Drawn Carriages to Delivery Drones People had been delivering goods for centuries using animals like horses and camels. Later, trains, ships and cars sped things up. But the last mile - getting items from large vehicles to destinations - remained difficult. E-commerce boomed in the late 1900s, creating demand for fast package shipping. FedEx and UPS rose with planes, trucks, and vans worldwide. Yet online orders grew so much that last-mile challenges also increased. Customers wanted speedier, cheaper, flexible options while costs rose for companies. To meet these problems, firms tried new methods. with free two-day shipping. They also did same-day and one-hour delivery, using instead of employees. Amazon has also experimented with drones, robots and lockers too, seeking better last-mile solutions. Amazon launched Prime in 2005 independent drivers The Last Mile: The Final Frontier of Logistics Last-mile delivery is important for several reasons. First, it is a key factor in customer satisfaction and loyalty. According to a survey by McKinsey, 25% of online shoppers would switch to another retailer if they had a bad delivery experience. Moreover, 56% of millennials said they would pay more for same-day or instant delivery. Therefore, offering fast and reliable last-mile delivery can give a competitive edge to online retailers. Second, last-mile delivery is a major cost driver in logistics. According to some estimates, last-mile delivery accounts for 41% of the total delivery cost and 53% of the total carbon emissions. Reducing the cost and environmental impact of last-mile delivery can improve the profitability and sustainability of online retail. Third, last-mile delivery is a source of . As technology advances and consumer behaviour changes, fresh solutions and players emerge in the last-mile ecosystem. For example, startups like DoorDash, Instacart, and Postmates offer on-demand delivery services for food and groceries. Platforms like Uber Eats, Doordash, and Grubhub connect restaurants with customers and couriers. Companies like Yandex, Starship Technologies, and Nuro develop autonomous vehicles for last-mile delivery. innovation and opportunity The Tech Titans of Today: Yandex, Amazon, and Beyond Two companies that stand out as leaders in last-mile delivery innovation are Yandex and Amazon. Yandex is a Russian internet giant that provides services like search, advertising, e-commerce, maps, cloud computing, and self-driving technology. They've been developing autonomous vehicle technology and currently have a fleet of . These cars have driven over 4 million miles on public roads in Russia, Israel, and the US under various weather and road conditions. With this level of maturity and real-world experience, Yandex is considered one of the global leaders in self-driving development. since 2017 170 self-driving cars Yandex also launched their own delivery robot called Yandex.Rover for transporting small and medium packages autonomously. This robotic option could help automate last-mile delivery within their ecosystem and beyond, providing a sustainable way to deliver food and goods. In total, Yandex has invested approximately into their self-driving division. Recently, they launched Yandex.Rover commercial operations in Skolkovo, a major Russian tech hub. Yandex plans to introduce their autonomous services in Ann Arbor, Michigan and Tel Aviv in the near future as well. $150 million Amazon is another dominant player in e-commerce and technology, known for online retail, streaming, cloud services, AI, and smart devices. They've invested heavily in logistics to speed up and improve delivery. Amazon developed their own delivery network called Amazon Logistics which uses stations, sorting centres, transportation providers, and delivery partners to ship packages independent of traditional carriers. Amazon has also introduced autonomous solutions like , which aims to deliver packages by drone within 30 minutes, and Amazon Scout, a six-wheeled sidewalk robot for navigating neighbourhoods and dropping packages at doorsteps. Other experimental delivery methods from Amazon include in-car/home delivery with and locker/counter pickup with Amazon Hub. Through technological capabilities and their massive customer base, Amazon has become a major force in last-mile delivery. Amazon Prime Air Amazon Key Beyond Yandex and Amazon, companies like , , and are innovating as well. Starship designs small self-driving delivery robots for food, groceries and more across Europe and North America. Nuro partners with retailers to autonomously deliver goods like groceries and prescriptions using their vehicles. And Zipline runs what they claim is the world's largest drone delivery network, transporting medical supplies in Africa and beyond. In the next section, we'll examine some current last-mile delivery case studies. Starship Technologies Nuro Zipline Case Studies from the Front Line: Last-Mile Delivery Today Yandex.Rover in Serbia In June 2023, , Serbia, marking the first time it operated outside of Russia. Yandex partners with local restaurants, cafes, and supermarkets to offer fast, convenient food, grocery, and item delivery to customers via the Yandex app. Customers can track the robot's location and estimated arrival time. Using cameras, lidar and GPS, Yandex.Rover autonomously navigates sidewalks, crosswalks, and traffic lights. Speakers and a screen allow it to communicate with pedestrians and drivers. Its cargo compartment only opens with a unique QR or PIN code provided to the customer. Yandex launched its delivery robot Yandex.Rover in Belgrade This launch was part of Yandex's European expansion strategy. As a leading provider of search, maps, taxi, and food delivery services in Serbia, Yandex aimed to highlight its technological capabilities and offer a unique value with Rover as well. Amazon Prime Air in US, UK, and Italy Amazon Prime Air will integrate drone deliveries into Amazon’s existing fulfilment network, deploying from Same-Day Delivery sites. Amazon customers in Italy, the UK, and an additional U.S. city will soon have the option to get their packages delivered by a drone beginning in late 2024. The Prime Air drone delivery system is expanding to international locations and adding a third city in the United States by the end of next year— specific cities in the U.S. and abroad will be named in the coming months. Zipline in Nigeria In October 2022, through partnerships with the Kaduna State Government and Gates Foundation. Zipline delivers blood, vaccines, medication, and medical supplies from its Kaduna distribution centre to over 1,000 health facilities across the state via 120 km/h flying drones carrying 1.8kg payloads within 30 minutes in all weather. Zipline launched drone services in Nigeria Zipline's Nigeria launch expanded its mission begun in Rwanda in 2016 and Ghana in 2019, where it delivered over 1 million vaccine doses and saved thousands of lives. It aimed to further support universal healthcare in Nigeria. Upcoming Technologies: What’s In the Pipeline? Technology is one of the main drivers of innovation and disruption in last-mile delivery. As technology advances, novel solutions and possibilities emerge that could enhance the efficiency, convenience, and sustainability of last-mile delivery. Here are some of the upcoming technologies that could have a significant impact in the coming years: Hyperloop is a proposed mode of transportation that involves sending pods or capsules through a network of tubes or tunnels at shockingly high speeds, up to 1,200 kilometres per hour. Hyperloop could enable fast and low-cost transportation of goods over long distances, reducing the need for trucks, planes, or trains. Hyperloop could also connect urban areas with rural or remote areas, expanding the reach and accessibility of last-mile delivery. : Hyperloop Blockchain is a distributed ledger technology that enables secure and transparent transactions among multiple parties without intermediaries. Blockchain could improve the traceability, accountability, and efficiency of last-mile delivery by enabling real-time tracking of goods, smart contracts for payments and deliveries, and decentralized platforms for peer-to-peer delivery. : Blockchain 3D printing is a technology that enables the creation of physical objects from digital models by depositing layers of material. 3D printing could revolutionize last-mile delivery by enabling on-demand and localized production of goods, reducing the need for inventory, warehousing, and transportation. 3D printing could also enable customization and personalization of goods, enhancing customer satisfaction and loyalty. : 3D Printing Biotechnology is a technology that involves the manipulation of living organisms or biological systems. Biotechnology could transform last-mile delivery by enabling novel solutions for packaging, preservation, and delivery of goods. For example, biodegradable packaging made from algae or fungi could reduce waste and environmental impact. Bioengineered foods or medicines could extend shelf life and improve quality. Biosensors or biometrics could enable secure and convenient delivery verification. : Biotechnology However, challenges exist - technical and regulatory hurdles, social acceptance, ethics, and cybersecurity risks require careful evaluation before adopting unfamiliar concepts. Strategic Shifts: Predicting the Unpredictable While technology drives change, unpredictable human factors also impact last-mile delivery's future. Shifting consumer behaviours, economic conditions, environmental concerns, politics, and natural disasters create uncertainty, with opportunities and risks. To adapt amid this complexity, companies must strategically pivot. Transitioning focus from products alone to customers' experiences demands deep understanding of needs, personalization, and loyalty-building through positive impact. Transitioning from ownership to access alters business models, as shared subscriptions and rentals replace individual purchases in the circular sharing economy. Platform ecosystems boost availability. Moving operations away from centralization delegates control and fosters innovation through distributed localized partner networks, peer-to-peer crowdsourcing, and community cooperation. However, achieving the right product-service balance with these shifts requires trade-offs. Implementing changes across dispersed stakeholders introduces risks that must be diligently governed. Incentives and resource coordination become more intricate. Careful consideration weighs consequences before wholesale adoption. But navigating humanity's disruptive course demands agility and vision. Strategic flexibility positions resilience amid uncertainty, preparedness navigating both risks and opportunities ahead. With responsibility and diligence, last-mile leaders can help deliver the future people need. The Future is Now One thing is clear - we are witnessing the dawning of a new era. An era where imagination meets innovation, where science fiction becomes a science fact, where the impossible becomes possible. Yesteryear's visions of drones, robots and self-driving vehicles delivering goods on demand have already transformed into today's reality. Pioneers like Yandex, Amazon and Starship have pulled the future into the present and set us boldly on a course we could only dream of just a decade ago. What will the next decade bring? It is impossible to say for certain, but if history is any guide, it may surpass even our most outlandish speculations. The innovations will continuously reshape life as we know it, delivering value to all in their wake. As for the rest of us, our role is clear - embrace change with optimism and curiosity. Keep an open mind for what is to come, everyone!

The Last-Mile Delivery Revolution: What’s Next on the Horizon?

The rise of e-commerce has been remarkable over recent years, and it shows no signs of plateauing. As this sector expands, the demand for streamlined delivery systems also grows significantly. Central to this demand is the idea of last-mile delivery, a term initially used in the telecom sector and then adopted by logistics and supply chain management. Last-mile delivery is the last step in the delivery process. It's the journey a product makes from a distribution center or facility to the final destination, usually a customer's home. Even though it's the concluding phase, last-mile delivery can be the most intricate and costly in the entire supply chain. Thus, it accounts for a substantial part of the total delivery cost. Factors such as traffic congestion, misaligned delivery timings, and increasing customer demands for speed and flexibility contribute to it vastly. Considering the influence of these obstacles on operational proficiency and customer contentment, companies are constantly exploring innovative strategies to make this process more efficient and cost-effective. With technologies ranging from artificial intelligence to self-driving vehicles, last-mile delivery is undergoing a transformative phase, becoming more streamlined and customer-focused. Challenges in last-mile delivery While last-mile delivery presents an opportunity for e-commerce businesses to distinguish themselves from competitors, it also has a unique set of its own troubles. One of them is meeting the growing customer expectations. Today's consumers demand not only speedy delivery but also the flexibility to alter delivery times and locations, which can complicate logistics planning. Plus, customers often expect real-time updates on their orders, which requires sophisticated tracking systems. Geographical issues also pose serious challenges. Goods delivery to outlying suburban and rural locations, where residences are widely dispersed, can be both time-intensive and costly. Meanwhile, deliveries in cities face their own unique challenges such as heavy traffic, scarce parking, and restricted access, which all can cause delays. Cost management is another big problem. With fuel, labor, and vehicle maintenance costs steadily rising, keeping last-mile delivery affordable without compromising on efficiency is a balancing act. Moreover, the environmental footprint of logistics has gained significant public attention, regarding emissions and sustainable practices. Safeguarding the delivered goods, especially when handling high-value or perishable items, is another aspect companies must handle. And finally, scaling last-mile delivery operations to accommodate increasing volumes of orders while maintaining the same level of service quality is a constant struggle for many companies. Handy tech solutions Innovative technologies have the potential to reshape last-mile delivery, helping to address the challenges we've outlined earlier. Today, there’s an array of technological options that a business can integrate into its day-to-day operations. Autonomous Vehicles Several companies are already deploying drones for delivery. In the US, Amazon, with its Prime Air service, plans to deliver packages up to five pounds in 30 minutes or less using unmanned aerial vehicles. Alphabet's Wing has also received regulatory approval to start drone delivery services in several regions of Australia, Finland, and the US. Ground robots are another nice development. For example, Starship Technologies has deployed its delivery robots in multiple cities worldwide, including the US, the UK, and the company’s homeland of Estonia. They operate autonomously 99% of the time and can carry items within a 4-mile radius. The company operates AI and Machine Learning Route optimization software, such as Routific or ORTEC, use AI and machine learning algorithms to calculate the most efficient delivery routes. These systems consider various factors, such as traffic, weather conditions, and package destinations, to reduce travel time and costs. These advanced systems evaluate a lot of factors when plotting routes. They take into account real-time traffic data, weather conditions, package destinations, and even the capacity of the delivery vehicle. Importantly, these systems are dynamic. They continuously learn and adjust to changing conditions, ensuring the most up-to-date information is used for decision-making. For instance, if there's a sudden traffic jam due to an accident, the system can quickly re-route the delivery vehicle to avoid delays. Internet of Things IoT technology is being used to create smart logistics networks. Companies like Roambee provide real-time visibility into the delivery process through sensor data. Their devices monitor location, temperature, humidity, and more, providing crucial information for deliveries, especially for sensitive goods. For example, pharmaceutical products or perishable foods need to be kept at specific temperatures, and real-time monitoring allows companies to ensure these conditions are maintained throughout the whole delivery journey. Moreover, IoT-enabled predictive maintenance can alert companies about potential issues with their delivery vehicles before they result in costly downtimes. This information, derived from continuous monitoring and data analysis, allows companies to manage their fleet, proactively. Blockchain Technology VeChain, a blockchain platform, is an excellent example of using blockchain in logistics management. This tech enhances transparency and security by offering a full view of the supply chain. So, consumers can use VeChain to verify product authenticity, and businesses can improve logistics processes with it. Visibility provided by such services helps companies identify any bottlenecks or inefficiencies in their logistics network, which leads to more streamlined operations and enhanced performance. Plus, the security of blockchain technology means that this data is less susceptible to tampering or fraud. Real-life examples Many companies have embraced innovative technologies to improve their last-mile delivery operations. Here are a few examples from worldwide leading businesses. FedEx This global delivery provider has been exploring autonomous delivery solutions for some time, and one result is Roxo, the SameDay Bot. It’s a delivery robot designed to transport smaller shipments, specifically in urban environments where traffic and parking can be tricky. Roxo can navigate pedestrian paths and even climb stairs, which makes it an efficient option for quick, local deliveries. DHL Another logistics giant has leveraged AI and machine learning to optimize their delivery routes. The company has reported reduced delivery times, fuel costs, and CO2 emissions. Furthermore, DHL uses IoT for predictive vehicle maintenance, which allows them to improve efficiency by reducing unexpected downtime. Domino's Pizza The major fast food chain has partnered with Nuro, a leading autonomous delivery company, to use their autonomous vehicles for pizza delivery in the US. By providing customers with a unique pin to retrieve their order from the vehicle, Domino's has streamlined its delivery process and enhanced customer experience. Zipline Zipline is a company that uses drones for last-mile delivery of medical supplies. It operates in hard-to-reach places like rural Rwanda and Ghana, where conventional delivery methods just don’t work. By deploying drones, Zipline ensures that hospitals and health centers receive necessary supplies promptly. Potential problems Despite all these advantages, there are potential challenges that businesses need to navigate while using new tech solutions for their last-mile delivery. Regulatory issues may be a significant roadblock, especially concerning autonomous vehicles and drones. Different regions have various rules regarding unmanned vehicle operation, and compliance is important here. Cybersecurity is another concern. With businesses growing increasingly dependent on digital solutions, the threat of cyberattacks grows high. It's crucial to have strong security protocols to protect sensitive data, like customer details and payment information. The initial cost of these technologies can be significant. Deploying state-of-the-art AI systems, autonomous vehicles, or blockchain solutions requires a sizable investment. Some businesses, especially smaller ones, might find these upfront costs too much. And while some customers appreciate the convenience and speed these technologies offer, others may have privacy or safety concerns, particularly around drone deliveries and data sharing. Lastly, technical issues and the need for skilled personnel to manage these new technologies can also present challenges. Companies need to ensure they have the necessary technical expertise on hand to implement and operate these solutions. What lies in the future Looking ahead, new technologies are set to continue transforming the last-mile delivery sector. Innovations such as hyper-local fulfillment centers and advanced predictive analytics could lead to even faster and more efficient deliveries. We might also see wider adoption of blockchain for logistics, enabling even greater transparency and security in transactions. As technologies mature, we will likely witness increased cooperation between regulatory bodies and businesses to create supportive frameworks for innovations like drone delivery. This could make last-mile delivery even more efficient. The evolution of e-commerce logistics depends on our willingness to adopt these innovative technologies. Given the clear benefits — enhanced efficiency, cost-effectiveness, and customer satisfaction — it appears that businesses should be keen to invest in these advancements and utilize them. However, embracing these solutions should also come with a degree of caution. It's essential to navigate potential challenges, such as regulatory hurdles and cybersecurity risks, with due diligence. The right balance will allow businesses to optimize their last-mile delivery services effectively while maintaining customer trust.

Innovative Technologies for Last-Mile Delivery in E-commerce Logistics Scaling

138 Stories To Learn About Startups Top Story

123 Stories To Learn About Healthtech

Let's learn about via these 28 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Drones /Learn Repo [1. Drones: The Good, The Bad, The Ugly](https://hackernoon.com/drones-the-good-the-bad-the-ugly-cba73y05) With great power comes great responsibility. 2. IT Arena 2022 - Ukrainian Tech Braves the War This was my third invitation to attend IT Arena in Lviv, Ukraine. But this was the first time, and indeed a first for me overall that it happened during a war. 3. How Drones Are Transforming Big Data Analytics The world is transforming right before our eyes. We’ve heard about drones for a long time now, especially with big companies like Amazon using them for more efficient package delivery, a major trend in modern e-commerce. Instead of your local delivery man, a drone may drop a package right on your doorstep. The true power of drones goes well beyond that, though. They provide businesses with data that’s difficult to collect otherwise. In addition to taking aerial photos and videos, drones can collect information about everything from the health of crops to thermal leaks in buildings. 4. How Runaway Drones Close Airports (and Why We Should Put SIMs in Them) Drones go missing despite the state-of-the-art technology and equipment that's put into them. Here I explore a solution that could prevent such incidents. 5. How Zipline Drones Can Save Lives in Rural Rwanda Everything is easily accessible when your home is in a city. 6. World’s First Indoor Drone Combats COVID-19 And Delivers 99% Disinfection Rate The Aertos drones are able to fly stably indoors without external reference like GPS 7. Drones Will Change The Insurance Industry It is never too early to be thinking about the future. And if you are thinking about the future, think about drones. 8. 3 Books to Top the List on Digital Trends in 2021 3 Books to Top the List on Digital Trends in 2021. 9. Building a Raspberry Pi + ESP32 Drone: My First Steps Into Robotics A journey from a web developer to a drone maker, with all of the problems and solutions that come with making a drone using a Raspberry Pi and ESP32. 10. Drone Combats COVID-19 with UV-C to Achieve 99% Disinfection Rate Indoors We are entering a time of intense innovation and will likely see a lot of it in the months ahead. It should come as no surprise that we will see tech developments, especially in healthcare from all parts of the world. Just yesterday, a young boy created 3D printed "ear guards" to alleviate ear pain experienced by healthcare workers who must wear masks for long hours. The thinkers, the doers and the helpers will expedite advancements and guide us through this time. 11. Drones and Why They Have so Much More Potential Hearing the word “drone”  is likely to spark a picture of a weirdly-shaped, helicopter-alike small flying machine in almost everyone’s minds. Although this might be true, Unmanned Aerial Vehicle technology(UAV), widely acknowledged as drone technology is becoming more popular and diverse than ever before. Since its early development attempts by the French and American military services during World War II, it has undergone an unprecedented transformation to become what is so commonly used in the modern world. 12. How Drones Save Time and Money on Construction Sites How to build more cost-efficiently with drones, what drones do better than humans and where can you save money by using a drone on a construction site? 13. Small Object Detection in Computer Vision: The Patch-Based Approach How to carry out small object detection with Computer Vision - An example of finding lost people in a forest. 14. 'At The Forefront Of A Revolution': A Chat With Drone Maker Wingtra In 2014, Wingtra was founded as a professional drone manufacturer for the mapping and surveying industries. 15. How Agricultural Holdings and Farmers Can Use Drones for Precision Agriculture There are a bunch of things you can monitor on your farm with drones. 16. Everything You Need to Know About 5G 5G is the latest network connectivity experience that you can get. It comes along with a variety of benefits and possibilities. In fact, you can use 5G to achieve things that you couldn’t do with the help of 4G. Therefore, it is worthy to take a look at 5G and get a better overall understanding about all the possibilities that it can offer. 17. This Ladybug-Inspired Super Drone Buggy Self-Rights Itself After a Fall This bug-inspired super drone that won't need your help to get back to flying after it falls in environments that cannot provide human assistance to right it 18. Facial Recognition On Drone A facial recognition demonstration using Keras, Tensorflow, python and the drone Tello, from DJI. 19. Pleasure and Business Merge, Here Are 5 Industries Drone Tech Is Transforming It used to be that when someone heard the term “drone” the first thing they thought of was militaries flying unmanned aerial vehicles (UAV) over hostile territory to record valuable information or potentially even deploy weapons. With the advent of recreational drones, the focus has shifted, with many associating drones with fun, outdoor activities. 20. Understanding the Impact of Artificial Intelligence on Agriculture According to recent projections by the UN: 21. The Rise of Drones in Different Industries Description: Whether you follow technological developments or not, you cannot escape from hearing about drones. Drone technology is developing at the speed of light and they are becoming increasingly popular. Nowadays almost everyone wants to have a drone with a camera for personal or business reasons. 22. 5 Technology Trends For Construction In 2020 The construction space has always been behind in adopting technological advancements. Why? I'm not too sure either. These technologies I'm sharing may or may not seem new to you but recently, they are rapidly being adopted in construction. 23. 8 Amazing Camera Drones For Extraordinary Photos 1.DJI MAVIC AIR 24. Virtual Property Tours: A Tech Solution to Self-Isolation Technology is already being leveraged and used in a myriad of different ways in the property industry. Adapted from fintech solutions, proptech platforms (such as those used by the likes of Zoopla, Rightmove, etc.) that have been developed over the last few years have revolutionized the industry, connecting tenants, investors and developers with the right properties for them in way that wasn’t previously possible. 25. Top Drone Video Editing Software Without a doubt, assuming you have such expert gear for shooting as a drone, you have a ton of extraordinary shots in stock that simply should be made impeccable. They are diamonds in the rough waiting to be discovered after editing and completing a hypnotizing video. 26. The Black Swan: The Aircraft with No Pilots on Board The Black Swan is a cargo-hauling aircraft with no pilots on board. 27. The Coming Age of Killer Machines The most terrifying film of the year didn’t come from Hollywood. 28. Last-Mile Delivery: What Industry Leaders Look for in Startup Partners Major players in the delivery industry - including incumbents such as UPS, FedEx, SF Express and DHL - have demonstrated a keen interest in establishing relationships vertically with accomplished startups specializing in autonomous and intelligent logistics and transportation. The convergence of several key trends serves as the impetus for the collaborations: Thank you for checking out the 28 most read stories about Drones on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

28 Stories To Learn About Drones

has the potential to revolutionize the way we approach and address global challenges. As an experienced AI developer and a thought leader in the area with a proven track record of building and deploying AI products in both industry and academia, I have witnessed firsthand the transformative potential of AI. Artificial intelligence (AI) One area where AI can make a significant impact is in promoting environmental, social, and governance (ESG) practices. ESG factors have gained prominence in recent years, with investors, companies, and governments recognizing their importance in building a sustainable future. How can AI be used to promote ESG practices? Let's explore real-world examples to illustrate the potential of AI-driven solutions. AI For Environmental Management AI can offer innovative ways to monitor, analyze and reduce our impact on the environment. AI-driven climate models can provide more accurate predictions and help policymakers design effective strategies to mitigate the effects of climate change. DeepMind's AI system has in improving the accuracy of weather forecasts, which can contribute to better climate modeling. Climate change modeling. shown promise Satellite imagery analysis can detect illegal logging and deforestation in real-time, enabling rapid intervention. Global Forest Watch, for example, uses AI to and identify patterns of forest loss, which helps conservation organizations take timely action. Monitoring deforestation. analyze satellite images Energy consumption can be optimized in buildings and industries, reducing greenhouse gas emissions. For example, Google works with DeepMind to of its data centers by 40%, significantly lowering its carbon footprint. Energy management. reduce the energy consumption AI For Social Responsibility AI can contribute to social responsibility by promoting inclusiveness, fairness, and ethical behavior. Hiring and promotion data can be analyzed to identify and correct potential biases. Unilever has adopted to screen candidates fairly and objectively, which helps ensure a more diverse and inclusive workforce. Fighting discrimination. AI-driven recruitment tools Healthcare solutions can use AI to improve access to quality healthcare for underprivileged communities. Zipline, a drone delivery company, to remote areas in Rwanda and Ghana, significantly improving access to life-saving treatments. Health and well-being. delivers medical supplies Fintech solutions can use AI to provide access to affordable financial services for unbanked populations. Mobile lending platform Tala uses AI algorithms to , enabling it to provide microloans to people who would otherwise be excluded from traditional financial services. Financial inclusion. assess creditworthiness AI For Governance And Transparency AI can help improve governance and transparency by automating routine tasks, detecting fraudulent activities and ensuring compliance with regulations. Potential regulatory breaches can be detected by automatically monitoring and analyzing vast amounts of data. Ayasdi helps banks and ensure compliance with anti-money laundering regulations. Regulatory compliance. detect suspicious activities AI-driven solutions can streamline public sector processes and improve service delivery. The U.K.'s National Health Service (NHS) has begun , freeing up staff time for more critical responsibilities. Public sector efficiency. automating routine administrative tasks AI can analyze corporate governance data to assess a company's ESG performance and identify areas for improvement. Datamaran can help companies and opportunities, providing insights to enhance their sustainability strategies. Corporate governance. track ESG risks Challenges And Ethical Considerations While AI can drive ESG practices, it's crucial to address the challenges and ethical considerations associated with AI implementation, such as algorithmic bias, job displacement, and accountability. However, two of the most pressing challenges that we should focus on right now are data privacy and environmental impact. Data Privacy As AI relies heavily on data, it is essential to ensure that data collection, storage, and processing adhere to privacy regulations and respect individual rights. One way to address this is by implementing robust security measures within your organization to protect data from unauthorized access, tampering, or theft while concurrently providing training and resources for employees to understand the importance of data privacy. You can also use privacy-preserving AI methods such as federated learning in your projects, which allow AI models to learn from data without directly accessing sensitive information. We use such mechanisms to train AI models to reduce food waste based on consumers' data without actually accessing them. Also, don't forget to conduct periodic privacy impact assessments and security audits to identify potential risks and vulnerabilities in data-handling practices. It is also crucial to ensure that data processing within the company complies with data protection laws and regulations such as GDPR in the European Union and CCPA in the United States. Environmental Impact AI systems, particularly large-scale models, can consume significant energy, contributing to carbon emissions. To mitigate this, focus on the development and implementation of energy-efficient AI algorithms that require fewer computational resources and consume less power. You can also employ techniques such as model pruning, quantization and knowledge distillation to reduce the size and complexity of AI models. Moreover, opt for energy-efficient hardware such as specialized AI accelerators or low-power processors, which can help reduce the energy consumption of AI. Moving forward, also make sure to encourage and support research in sustainable AI, including the development of energy-efficient algorithms/hardware and privacy-preserving techniques. Collaborating with academic institutions and industry partners can help drive innovation in this area. Conclusion AI has the potential to transform the way we approach and address ESG issues, driving sustainable development and fostering a more equitable and just society. By leveraging AI-driven solutions in environmental management, social responsibility and governance, we can make significant strides toward a sustainable future. However, it is essential to recognize the challenges and ethical considerations associated with AI implementation. By addressing these concerns and adopting a responsible and ethical approach to AI deployment, we can harness the power of AI to promote ESG practices and contribute to the long-term well-being of our planet and its inhabitants. Also published here. Lead image . source

Enhancing ESG Initiatives with Artificial Intelligence

This tutorial demonstrates how to use C/C++ code in your KMM shared module. You'll learn how to embed a simple SHA-256 implementation on C-language in the KMM library and use it for iOS and Android target platforms. Kotlin Multiplatform Mobile (KMM) is an SDK designed to simplify the development of cross-platform mobile applications. You can share common code between iOS and Android apps and write platform-specific code only where it's necessary You can use C and C++ code both with Android and iOS . KMM utilizes this functionality and allows you to use C and C++ code in your shared module. (Xcode supports C, C++, and out-of-the-box, and Android has ) Android NDK It can be useful in app development; there are a lot of libraries that have already been written in C/C++. A great example of this technique is the which embeds the engine in the KMM code. Zipline project QuickJs In this article, we won’t talk about how it works under the hood and will instead concentrate on writing the code and setting up the Gradle build. The code for this article could be found in the repository Start Building a Library for iOS and Android We won’t talk about how to create a KMM library as it is very well explained in the official Kotlin tutorial. Let’s take the SHA-256 code written on C as an example, and copy it into our folder. The sample code for the SHA-256 implementation was taken from this . native/sha256 repository Because Android and iOS applications support C-code differently, we need to use the . expected/actual approach First of all, let’s set up our common part: /**
 * Encoder interface which can encode byte arrays to Sha256 format.
 */
interface Sha256Encoder {
    fun encode(src: ByteArray): ByteArray

    fun encodeToString(src: ByteArray): String {
        val encoded = encode(src)
        return buildString(encoded.size) {
            encoded.forEach { append(it.toUByte().toString(16).padStart(2, '0')) }
        }
    }
}

expect object Sha256Factory {
    /**
     * Creates a new instance of [Sha256Encoder]
     */
    fun createEncoder(): Sha256Encoder
} Then, continue to platform-specific implementations. Embed C-code for iOS Target Platforms For iOS target platforms, we will set up the Gradle plugin, which will build our C-code as LLVM bitcode. C Klib C Klib is an experimental library with limited support from the authors. All we need to do is provide a path to the source folder with C-code; in our example, it will be : native/sha256 cklib {
    config.kotlinVersion = libs.versions.kotlin.get()
    create("sha256") {
        language = C
        srcDirs = project.files(file("native/sha256"))
    }
} Then we need to create a file for the KMM tool which generates Kotlin bindings from a C-header file. .def cinterop To learn more about , you could read .def the official cinterop tutorial In our case, we just need to create our in the folder (default searching place for tool) and name package where Kotlin bindings will be: sha256.def nativeInterop/cinterop cinterop package = com.ttypic.clibs.sha256 Finally, we need to add the cinterop step for our target platforms and specify the header file: kotlin {
    // ...
    listOf(
        iosX64(),
        iosArm64(),
        iosSimulatorArm64()
    ).forEach {
        it.compilations {
            val main by getting {
                cinterops {
                    create("sha256") {
                        header(file("native/sha256/sha256.h"))
                    }
                }
            }
        }
    }
    // ...
} After resyncing our Gradle build, we will see Kotlin bindings in the package, and we are ready to invoke native code. Our implementation will look like this: com.ttypic.clibs.sha256 actual actual object Sha256Factory {
    actual fun createEncoder(): Sha256Encoder = NativeSha256Encoder
}

object NativeSha256Encoder : Sha256Encoder {
    @OptIn(ExperimentalUnsignedTypes::class)
    override fun encode(src: ByteArray): ByteArray =
        memScoped {
            val ctx = alloc<SHA256_CTX>()
            sha256_init(ctx.ptr)
            val srcPointer = src.toUByteArray().toCValues().ptr
            sha256_update(ctx.ptr, srcPointer, src.size.toULong())
            UByteArray(32).apply {
                usePinned {
                    sha256_final(ctx.ptr, it.addressOf(0))
                }
            }.toByteArray()
        }
} That’s it; now we can run tests to check if everything is OK: class NativeSha256Test {

    @Test
    fun `should pass library sha256 checks`() {
        checkEncodeToString("Kotlin", "c78f6c97923e81a2f04f09c5e87b69e085c1e47066a1136b5f590bfde696e2eb")
        checkEncodeToString("abc", "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
        checkEncodeToString("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1")
        checkEncodeToString("aaaaaaaaaa", "bf2cb58a68f684d95a3b78ef8f661c9a4e5b09e82cc8f9cc88cce90528caeb27")
    }

    private fun checkEncodeToString(input: String, expectedOutput: String) {
        assertEquals(expectedOutput, Sha256Factory.createEncoder().encodeToString(input.asciiToByteArray()))
    }

}

private fun String.asciiToByteArray() = ByteArray(length) {
    get(it).code.toByte()
} Embed C-code for Android Target Platforms For Android, we will be using . Android NDK uses under the hood; that’s why we need to provide the build file: Android NDK CMake CMakeList.txt cmake_minimum_required(VERSION 3.4.1)
# setup C standard we used
set(CMAKE_C_STANDARD 99)
# source code file mask
file(GLOB_RECURSE sources "../../native/*.c")
# build sources as dynamic library 
add_library(sha256 SHARED ${sources})
# link library
target_link_libraries(sha256) Then we have to specify this file in the Android plugin and setup the target platforms: android {
    // ...
    defaultConfig {
        // ...
        ndk {
            // target platforms
            abiFilters += listOf("x86", "x86_64", "armeabi-v7a", "arm64-v8a")
        }
    }
    // ...
    externalNativeBuild {
        cmake {
            path = file("src/androidMain/CMakeLists.txt")
        }
    }
} To invoke our native code, we need to use to map native code to JVM. Android Studio has great JNI support. Let’s create file, then include JNI and our library header files: JNI sha256-jni.c #include "jni.h"
#include "sha256/sha256.h" Now, we are ready to write our code: actual actual object Sha256Factory {
    actual fun createEncoder(): Sha256Encoder = AndroidSha256Encoder
}

object AndroidSha256Encoder : Sha256Encoder {
    init {
        System.loadLibrary("sha256")
    }
    external override fun encode(src: ByteArray): ByteArray
} It is very straightforward. First, we load our dynamic library, and next, we use to invoke native code. Now, all we need is to write JNI-bindings. You can do it yourself using the fully qualified class name or invoke the Android Studio context menu and choose JNI create JNI funtion JNI function in sha256-jni.c JNIEXPORT jbyteArray JNICALL Java_com_ttypic_clibs_AndroidSha256Encoder_encode(JNIEnv *env, jclass _, jbyteArray src) {
    BYTE hash[SHA256_BLOCK_SIZE];
    SHA256_CTX ctx;

    size_t len = (size_t) ((*env)->GetArrayLength(env, src));
    jboolean copied;
    jbyte* bytes = (*env)->GetByteArrayElements(env, src, &copied);

    sha256_init(&ctx);
    sha256_update(&ctx, (const BYTE *) bytes, len);
    sha256_final(&ctx, hash);

    (*env)->ReleaseByteArrayElements(env, src, bytes, JNI_ABORT);

    jbyteArray result = (*env)->NewByteArray(env, SHA256_BLOCK_SIZE);
    (*env)->SetByteArrayRegion(env, result, 0, SHA256_BLOCK_SIZE, (const jbyte *) hash);
    return result;
} And finally, we are ready to write tests: We need to write the instrumental test on the simulator; regular Unit-test won’t load our native library. class AndroidSha256Test {

    @Test
    fun should_pass_library_sha256_checks() {
        checkEncodeToString("Kotlin", "c78f6c97923e81a2f04f09c5e87b69e085c1e47066a1136b5f590bfde696e2eb")
        checkEncodeToString("abc", "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
        checkEncodeToString("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1")
        checkEncodeToString("aaaaaaaaaa", "bf2cb58a68f684d95a3b78ef8f661c9a4e5b09e82cc8f9cc88cce90528caeb27")
    }

    private fun checkEncodeToString(input: String, expectedOutput: String) {
        assertEquals(
            expectedOutput,
            Sha256Factory.createEncoder().encodeToString(input.asciiToByteArray())
        )
    }
}

private fun String.asciiToByteArray() = ByteArray(length) {
    get(it).code.toByte()
} We successfully embedded C-code in our Kotlin Multiplatform Library, which can be used in iOS and Android target platforms. Useful Links Github repository with final code Get started KMM Official KMM library tutorial Zipline project Native dependencies in KMM

How to Extend a KMM Shared Module With C/C++ Code

Trading Cup and Handles With MarketSmith Pattern Recognition

The Essential Architectures For Every Data Scientist and Big Data Engineer 

I've worked on teams building ML-powered product features, everything from personalization to propensity paywalls. Meetings to find and get access to data consumed my time, other days it was consumed building ETLs to get and clean that data. The worst situations were when I had to deal with existing microservice oriented architectures. I wouldn't advocate that we stop using microservices, but if you want to fit in a ML project in an already in-place strict microservice oriented architecture, you're doomed. In this article, I'll describe why microservice oriented architectures suck for machine learning. I'll then lay out how companies like AirBnB and Uber used a to manage it. feature store like StreamSQL Microservices have become the de-facto architectural choice for tech companies. Microservices allow small teams in large companies to build small, independent components. Teams can solve problems and meet requirements without having to retrofit into a giant monolith. It lets teams move faster. However, when overused, turning a user session token into a user profile can trigger twenty network calls. I saw this video on Twitter recently, and it inspired me to write that post. The video is hilarious and captures this concept perfectly: Why is it so hard to display the birthday date on the profile page? Product Manager First we have to call the Bingo service to get the UserID, then we call Papaya to turn it into a user session token. We validate those with LMNOP, then we can pull the user info down from Racoon. But Racoon isn't guaranteed to have [the birthdate], so next we call ...* Engineer Example: Reddit Recommender System Microservices are exceptionally painful when building machine learning models on complex data sources like user behavior. In these cases, to make a prediction, you need to pull from a multitude of microservices, that in turn pull from another multitude of services to get all the contextual information they need. For example, if you were to build a personalized reddit feed, you may want to know all communities a user is a part of, the top posts in these communities, any posts they've clicked on and liked, and more details. Adding additional inputs can provide more signal for the model to pick up on. For example, certain users may have drastically different behavior on the weekends than weekdays. Providing the model with the additional input - day of the week - allows it to pick up on those trends. Each logical input to a machine learning model is called a feature. The process of hypothesizing, building, and testing inputs to a model is called feature engineering. Typically, feature engineering is one of the most time-consuming tasks for ML teams. Creatively coming up with new features is only part of the process; most of the time is spent finding the data you need, learning its peculiarities and edge cases, and building data pipelines to clean and transform it into a usable form. In a Microservice-based architecture, the only way to collect certain data is via an API call. A single model may have a wide variety of features that require API calls. For example, the model may need to query the user service for the communities a user is a part of, then it needs to query the top posts in each of those communities, and then query the likes that each of those garnered. Since systems are almost never built for this sort of usage, it will set in motion a domino effect of downstream network calls. Unlike a web front end, ML models are not flexible around missing features. It will have to wait for all of the requests to complete or risk providing garbage results. ML models themselves are quite computationally heavy and very slow; mix this with microservices, and it becomes impossible to serve real-time recommendations. Getting Training Data Sucks Machine Learning models work by mimicking what they observe. To do this, a model requires a dataset of observed results and the inputs at the point in time of the observation to train on. Generating a training set in a microservice-based architecture sucks. In our Reddit example, we'd want an set of user upvotes along with the feature set at the moment they performed the upvote. Though it's far from ideal, we can technically get user upvotes and all post data by scraping the user and post microservices. However, point-in-time correctness is what makes this problem impossible in many cases. To train, we would need to know what rank the post was in the subreddit; however, the subreddit microservice is unlikely to support retroactive queries. One option is to avoid the microservices entirely by breaking encapsulation and reading from database dumps. We could now by-pass the APIs and plug the datasets directly into Apache Spark or some other batch processing system. We can join tables, and work with the data in a far more convenient way. It also makes it possible to generate training datasets as fast as Spark can crunch the data. While using a data lake seems reasonable at first, it leads to the ML services being dependent on the schema of the raw data. The schema will surely change over time. Microservices are retired and replaced. Data inconsistencies, errors, and issues pile up. Eventually, each ML team has to maintain data pipelines that have devolved into archaic messes of spaghetti code to bandaid the situation. The pipelines are fragile to changes and take a vast amount of time and resources. Image Source Another option is to utilize an event steaming platform like Apache Kafka, Apache Pulsar, or Segment to allow ML teams to subscribe to the event streams that they need. Many of the pitfalls for data lakes also apply to event streaming. However, unlike data lake dumps, event streams tend to have higher quality data. Since event streams often power mission-critical services, teams are held to a higher standard regarding data quality and documentation. Conversely, data lakes are exclusive to ML and analytics teams and are not held to a high standard. Event stream processing suffers from the cold start problem. Event streaming platforms are rarely configured to retain events for long, often it's for as little as one week. If you want to generate a new feature you may be stuck with only the last week of data to generate a training set. The cold start problem is further pronounced when creating a new stateful feature. A stateful feature requires you to aggregate events over some window of time. For example, the number of posts a user made in the last week. In these situations, it could take weeks to even start generating the training data set. Feature engineering is an iterative process. You generate a hypothesis, build an experiment, and run a test. You either merge it into the main model or scrap it. The faster iterations happen, the faster the model gets better. If it takes weeks to perform a single test, ML teams lose their ability to perform their jobs. Teams end up wasting away plumbing data pipes and playing politics to get access to data, rather than building better models. Image Source How Teams are Solving the Microservice Issue The microservice problem isn't new or unique. Intriguingly, many companies independently landed on the same solution to the problem. AirBnB built Zipline, Uber built Michelangelo, and Lyft built Dryft. These systems are collectively called Feature Stores. What is a Feature Store? A provides a standardized way for data scientists to define features. The feature store handles generating training data and providing online features for serving. It abstracts away the data engineering from the ML workflow. Under the hood, it coordinates multiple big data systems to seamlessly process both incoming events and a past events. If you're interested in the specific technologies, [ feature store here's the layout of our feature store infrastructure. In the original "Reddit" microservice architecture, each service owned its own data. The posts microservice is the source of truth for data about posts, the user microservice is the source of truth for data about users, and so on. A feature store attempts to create its own views of this data in its own internal data structures. It does this by processing a stream of domain events into materialized state. Domain events are logical events such as when a user upvotes an article or when a new post is created. A materialized view is the result set from running a query on an event stream. So, if we wanted the model to know the number of posts that a user upvoted, then we could create a materialized view with this logic: SELECT user, COUNT(DISTINCT item) FROM upvote_stream GROUP BY user; All the materialized views exist in the same feature store and are preprocessed for ML use. We've merged all the microservice data that we care about into a monolithic data store. This eliminates the problems related to fetching real-time features from microservices. Now features can be fetched in one round trip. The feature store has the added benefit of being fault resistant, since the materialized views are stored in a highly-available and eventually-consistent data store. We own our own business logic in creating features since we are processing the raw events ourselves, so we are loosely coupled to the business logic in each microservice. Event Sourcing to generate Training Data Our features are kept consistent with incoming events. It's essentially a mirror of the microservices tables, but already preprocessed for use in ML and all in one place. Unlike the microservices, a feature store persists every incoming event indefinitely into a log. If we were to replay every single event in the log to a blank feature store, we would end up with the exact same materialized state. The event log becomes the source of truth for the system. This design pattern is called Event Sourcing. Event Sourcing lets us generate a training dataset for our models. To illustrate how this is done, let's take the Reddit example where we want to predict the next post that a user will upvote. The relevant domain events are streamed into the feature store, which then updates the model input features. Observed outcomes should also be streamed to the feature store, in this case every user upvote. feature_store.append_observation(userId, postId, now()) Since the feature store maintains a log of every event and the logic to turn an event stream into state, it can get the state of the features at any point in time. To generate a training set, it loops through the observations and generates the feature set at that point. By combining the two, it ends up with a training dataset. def generate_training_set(): observed observations:
    feature_store.process_events_until(observed.time)
    features = feature_store.get_features(observed.userId) (features, observed.postId) for in yield Design Considerations A Feature Store allows us to decouple from the microservice architecture, and own our own features. However, building and maintaining a feature store is not free. Teams should consider the following points before deploying feature store infrastructure. System-Wide Event Streaming A feature store requires that domain events are passed through an event streaming platform like Kafka or Pulsar. This allows the feature store to materialize its state independently of the microservices. Persisting the event log allows it to materialize the features at any point in time. Moving a large microservice-based system to use event streams is a monumental shift. Routines have to be injected to capture important events. This may require updating old, mission-critical microservices with a new dependency and new error conditions to catch. Another option is to use Change-Data-Capture semantics from each database to turn updates to a stream. However, the feature store is then vulnerable to schema changes inside a microservice's database. Handling Event Schema Changes A feature store is still dependent on the schema of the event streams. If a stream changes its schema, or a microservice misbehaves and uploads garbage data, it can incapacitate the feature store downstream. Event streams schemas should be treated with the same care as a database schema. Migration procedures should be clear and tested. Events should be written with an extensible format like Protobuf or JSON. Storage and Compute Capacity Processing and storing huge amounts of data is not free. In many cases, the feature store will repeat computations performed by individual microservices. A feature store trades infrastructure cost and complexity for developer velocity and ease of use. Building and maintaining a feature store requires both money and specialized engineers. Documenting and Sharing Input Features and Data Sources ML features are often applicable in many different use cases. Reddit may have many different ML-powered features that all use a user's activity to make decisions. Discovering and understanding features that others have built can speed up ML development time and provide inspiration in feature engineering. Since feature stores are a relatively new piece of architecture, teams will have to document how features should be published and shared. Real World Feature Stores There are a few example of internal feature stores built at large companies: , , and . We're building the next generation of hosted feature store at , check it out we ! AirBnB's Zipline Uber's Michelangelo Lyft's Dryft StreamSQL just entered beta (The author is the Founder at StreamSQL) Originally published here: https://streamsql.io/blog/microservices-with-machine-learning-feature-store

Why Microservices Suck At Machine Learning...and What You Can Do About It

With great power comes great responsibility. Drone technology has shown exponential growth in the past decade. In 2013, the concept of Amazon using drones to make deliveries seemed like an idea transplanted straight out of a sci-fi novel. Now, we live in a world where governments are able to use drones . Increasingly, we see many new startups that are launched every day with the intent of using drone technology to make the world a better place. to combat the spread of coronavirus But the history of drones has not been completely innocent. Even though drones have a myriad of productive uses, it’s arguable that drones were invented to be used in warfare. This article explores the light and dark sides of the incredible potential drones present to us. Here, our past shines a beacon into the future: where we have been, where we could go, what are the potential misuse cases of the technology and what we can do to prevent it. The Good: Access To The Inaccessible Conversation in the United States has been buzzing for years over the idea of using as a means of delivery for companies like Amazon and UPS. However, drones have found an unexpected niche in delivering medical necessities in countries around the world. Autel Evo 2 drones A few startup companies have been able to partner with governments in Africa to use drone technology to provide life-saving vaccines to remote villages in the region. Swoop Air, a Melbourne-based company founded by Eric Peck, cooperates with countries like Malawi, the Democratic Republic of the Congo, and Mozambique to deliver a range of medical products and services otherwise inaccessible in those parts of the continent. These are a wide range of medical products- vaccines; HIV/AIDs test kits; antibiotics like penicillin; antimalarial and anti-venomous treatments; and even payment for healthcare workers in those rural areas. As Eric Peck explained , “‘We quickly realised that although it was only a 20-minute flight across the island for us, it was a two-day walk to go across the top of the volcano or to get to some of the villages it was five hours on a boat.’” in an interview with ZDNet By increasing access, these companies empower healthcare workers in those regions to serve their staff and patients as effectively as possible. Swoop Air is not alone in its efforts. Zipline Inc. by working with the governments and hospitals in Rwanda and Ghana. has accomplished similar feats Other companies like Matternat, which operates within the United States and Switzerland, to provide a more cost-effective, on-demand delivery system for a time-sensitive specimen. use drones to collaborate with labs and pharmaceutical companies The Bad: Drone Warfare For all the good that drones have done for the world—for hobbyists, aeronautic researchers, businesses, and healthcare workers worldwide—they have also been employed as instruments of war. In fact, , dispatched in warfare as early as 1849. We have seen the devastating impacts of drone warfare in the past few years in places like Syria, Libya, and Turkey. the earliest drones were unmanned balloons carrying bombs But now, the Russian company Kalashnikov Group is developing another wartime application for drones — . Imagine the potential lethality of a kamikaze bomber with no pilot, intended to deliver devastating impact as cheaply as possible. suicide drones Their drone KUB-BLA, first announced at the 2019 IDEX trade show in Abu Dhabi, is designed with exactly that purpose in mind. The drone, , utilizes the advantages of “‘hidden launch, high accuracy of the shot, noiselessness and ease of handling’” in order to “‘destroy remote ground targets’”. as quoted in The National Interest Kalashnikov Group is not alone in this endeavor. The U.S. Air Force is rushing to create its own unmanned suicide drone to compete. It has awarded the contract for its Low-Cost Attritable Aircraft to the California-based drone company Kratos. Only time will tell how technological advancements in airstrike drones will impact warfare as we know it. The Ugly: Regulations Finally, the biggest limiter of drone potential comes down to regulations within individual countries. In the United States, drones’ applications for delivery are hampered by existing regulations established by the Federal Aviation Administration’s Part 107, . which requires the pilot to be able to see the drone they are piloting at all times This regulation has been a point of debate between parcel companies and the FAA for years. However, now the debate is shifting, as many argue that the U.S. would benefit in its fight against coronavirus if those regulations were lifted. Many theorize that the U.S. could follow the example of many other international governments like and who are using drones to catch people breaking quarantine, without putting additional stress on already-strained emergency response services. the UK Spain Some would insist that the U.S. regulations in place are crippling the country’s ability to make use of every tool in its arsenal to fight this pandemic. But there is a strong counterargument: once you let Pandora out of the box, you can’t shove it back in again. Once we begin using drones for surveillance—rather like we saw with the rise of the NSA—that surveillance will not go away once the threat of the pandemic is resolved. Moving Forward Drones are hardly a panacea for the world’s many challenges; for every solution drones provide, there is a cautionary tale waiting on the other side of the coin. However, fear of possible misuse of the technology should not cripple our innovation. Nor should our excitement for the future cause us to throw regulations out completely. What are your thoughts on the future use of drones?

Drones: The Good, The Bad,  
The Ugly

I think I’ve seen it: Kigali! I am lucky to have the opportunity to travel to many fascinating and exciting places around the world. This week I have been in Kigali, Rwanda. Wherever I go, entrepreneurship, innovation, and disruption are among the most discussed topics. The phrase “the Next Silicon Valley” is often used. And I hate this phrase. Everybody should know by now that Silicon Valley cannot be replicated. It’s a unique ecosystem, mindset, or — as some have even referred to it — religion. Whenever I visit Silicon Valley, you can breathe the “entrepreneurial spirit.” But I can also see that the shine of Silicon Valley starts to wear (as being described in many Medium stories recently). It’s becoming more “corporate,” profit-driven and many “world-changing” innovations are starting to happen in other places. However, it is still difficult to imagine an ecosystem as powerful or impactful. The usual suspects, such as Tel Aviv, Berlin, London, Toronto, Amsterdam, Stockholm, Tokyo don’t come close when it comes to the success and impact of “Silicon Valley.” Don’t get me wrong. The number of start-ups and investments are increasing in these regions. And, yes, you can see (and measure) the effects of innovation on business and job creation. But the entrepreneurial spirit isn’t as remarkable or as visible as what I’ve experienced in Silicon Valley. So, when I landed in Kigali, Rwanda last week, I had no expectations whatsoever. I really didn’t know what to expect. But after being here for one week, I experienced one pleasant surprise after another. Does this mean that Kigali and Silicon Valley can be compared in terms of the ecosystem architecture? No, of course not, the “ecosystems” are lightyears away from each other. It doesn’t make sense to compare them. But the excitement and energy I felt in Kigali was very similar to my first visit to Silicon Valley. I don’t say that Kigali will be the next Silicon Valley, but I have experienced something exciting, unique, and potentially transformative. During the week, I could see a distinctive innovation and entrepreneurial ecosystem emerging. The Kigali ecosystem has the potential to become extremely attractive to investors (foreign and domestic), attract regional talent, and have an enormous impact on the local society and the environment. The focus in Kigali is on “tech-innovations for good” in the crucial areas that the United Nations refers to as Sustainable Development Goals, instead of clicks on advertisements, e-commerce transactions, or attracting as many subscriptions as quickly as possible. So here are three ingredients that I found surprising and made me believe that we should all keep a close eye on the evolution of this particular ecosystem. Community thinking and partnering for innovation When I arrived, I could never have expected that the 2018 World Economic Forum reference to Kigali as the “cleanest city on the planet,” could have been accurate. But now I can testify that in terms of the lack of trash, as well as green initiatives, Kigali looks a lot like Singapore. But, apparently, without the need for strict (criminal) laws on littering. After having talked to residents, I found out that government initiatives led to well-maintained public transportation systems and road and communication infrastructures. But it was mentioned repeatedly that the main contributors to Kigali’s cleanliness are the residents themselves who are performing a day of community work, called “ ,” once a month. This “requirement” is mainly “enforced” through creating awareness, bringing residents from the informal to the formal economy by creating relatively well-paid jobs for cleaners, trash collectors and gardeners, but also by creating and setting up partnerships with local businesses (to establish trash collection sites in suburban areas and to install public toilets). umuganda This sense of community and commitment is also felt across initiatives for more disruptive innovation and entrepreneurship. Many mention the speedy and simple business registration process (note the 29th position in the World Bank’s Ease of Doing Business Ranking, which is higher than France, The Netherlands, and Israel). There is also an established community for early-stage tech start-ups, incubators, and free co-working spaces. This community has already attracted foreign entrepreneurs. For instance, when I was waiting in the lobby of the hotel, I saw two employees of Silicon Valley-based company Zipline. Zipline (according to their website) has built the world’s fastest and most reliable delivery drone. The company found in Rwanda a “laboratory-like” environment to pilot blood and medicine deliveries by drone. This kind of partnering-for-innovation (with a social and environmental impact) can be found everywhere in Kigali. It is only to be expected that we will soon see the further development of smart city initiatives, autonomous machines, and other emerging technologies for good. Younger generation While strolling through the streets of Kigali, I was struck by the number of young people. More research (back in the hotel) showed that forty percent of the Rwandan population is aged between 14 and 35 years. These tech-savvy “millennials” will help with the transition from an agriculture-driven and handicraft economy to a knowledge-based and digital economy. We often hear talk of how digital technologies can help agrarian economies “leapfrog” industrialization, but on the ground in Kigali, I saw that such talk can be very real. What is even more critical in this respect is that the younger generation puts significant pressure on established and traditional companies. For instance, during one of my workshops it became very clear that traditional banks are not trusted by the younger generation anymore. This trend makes it easier for innovative companies, products, and services to thrive. Also, people seem willing to quit their jobs to experiment and are not afraid to start something new. For instance, during one of the dinners a member of a Development Finance Institution told me about his afternoon meeting with two young entrepreneurs who successfully started a human resource management company specifically targeting the younger generation. Ordered chaos I am confident and excited about the potential of Kigali. That’s not to say that everything works efficiently. Of course, there is still some degree of “chaos” in this relatively young city (there were only 6,000 inhabitants in 1962) and it obviously has had a turbulent recent history. And, foreigners (referred to as “Mzungu”) are still viewed with some scepticism. But here is the thing. The city doesn’t seem to be consumed by tradition or history. It celebrates its lack of bureaucracy. It embraces the future and the possibilities of a digital age. The city can best be described as existing in a state of “ordered chaos.” This provides the freedom to be entrepreneurial and innovative (something that is not true in overly-ordered societies) but also has enough structure and infrastructure for such innovation and entrepreneurship to have the chance to flourish and succeed. This cocktail of qualities is already being acknowledged by foreign investors, such as sovereign wealth funds and development banks. In one of my conversations with a foreign investor last week, the term the new “Wild West” was mentioned. And not in a pejorative sense, but in the sense of exciting, new and unknown opportunities. I can’t wait to go back to Kigali and see how emerging technologies help the country (and perhaps the world) change for good. 👏 Thank you for reading! Please click the below, or leave a comment. There is a new story every week. So if you follow me you won’t miss my latest insights about how the digital age is changing the way we live, work and learn.

I hate to talk about the “Next Silicon Valley,” but…

50+ High Growth Companies to Launch Your Career With One of the most challenging aspects of growing up is figuring out how and where to spend your time. The world of “career advice” is incredibly noisy. Professors, friends, parents, irrelevant medium authors — everyone is telling you what you should and should not be doing with your life. The good news → You get to decide how and where to spend your time. The other goods news is that the world is increasingly massive…and there are so many exciting things you could do…infinite places you could go and learn… it is your life! But why do we graduate from college and settle for conventional jobs that do not unlock our real potential? — Kevin Hartz I put together this list of high-potential opportunities for people (of all ages!) who are interested in leveraging technology as a vehicle for high-velocity change and impact. what I think to be The list spans companies of all sizes tackling all sorts of problems (Finance, Healthcare, Social, Construction, etc.). Hopefully this list can be helpful as you make your next move! A few notes/disclaimers: Be sure to make your own judgements with any of the opportunities presented here and recognize that this list is not “an official ranking” of any sorts. It is not “backed by data” or any hard metrics whatsoever. This list is by no means de facto or comprehensive. Feel free to disagree with and question everything :) This is just one of many resources — be sure to also check out & . StartupSift Breakout List You will notice that many of these companies may or may not be hiring for your “specific role.” Others have thousands and thousands of hungry applicants. That is okay! You may have to cold email, create a position, or find other unique ways to stand out. You can do it! have thoughts? need help? i'd love to chat :) message me on twitter or here - jordan gonen 🌊 And without further ado…here is… The Signal: 50+ High Growth Companies to Launch Your Career With 🚀 (late 2018) Finance Blend (disclaimer: I interned at Blend) Blend’s technology empowers top mortgage lenders to be more efficient, compliant, and borrower-friendly. Brex Corporate cards for startups. Online application, no personal guarantee, no security deposit required, and built for modern businesses. Powered by the Visa network. Carta Carta helps private companies, public companies, investors, and employees manage their cap tables, valuations, portfolio investments, and equity plans. Cash Square is a merchant services aggregator and mobile payment company that aims to simplify commerce through technology. Square Cash is their mobile app (a Venmo competitor)…“the easiest way to send and receive money on the fly.” Coinbase Coinbase is an online platform that allows merchants, consumers, and traders to transact with digital currency. It allows its users to create their own bitcoin wallets and start buying or selling bitcoins by connecting with their bank accounts. Even “How busy people master their money.” We partner with employers to offer Even as a benefit so that employees can get paid on demand. Paystack Modern online and offline payments for Africa. Shogun Shogun Enterprises is a financial technology platform for home improvement. Stripe Stripe is the best software platform for running an internet business. They handle billions of dollars every year for forward-thinking businesses around the world. Titan The modern investment firm committed to exceptional returns and advisory through actively managed portfolios. Life is not a dress rehearsal — this is probably it. Make it count. Time is extremely limited and goes by fast. Do what makes you happy and fulfilled — few people get remembered hundreds of years after they die anyway. — Sam Altman Insurance / Legal Atrium Atrium builds products and services to help automate legal workflows, innovate new processes, and help firms identify new clients. ClearCover Clearcover is the smarter car insurance choice, offering better coverage for less money. DoNotPay DoNotPay is an online robot lawyer that allows anyone to automatically claim asylum in the U.S, U.K, and Canada for free. Lemonade Lemonade offers homeowners and renters insurance powered by artificial intelligence and behavioral economics. Newfront Newfront is the modern commercial insurance brokerage. No 20-page applications, confusing policy terms, or painful renewal cycles. Newfront’s experienced brokers and technology remove the headache from business insurance. Root Root is a mobile-first insurance carrier that uses an app to rate drivers based primarily on how they actually drive — not who they are. “It is the ultimate luxury to combine passion and contribution. It’s also a very clear path to happiness.” — Sheryl Sandberg HR Gusto Gusto provides a cloud-based payroll, benefits, and human resource management solution for businesses. Humu Humu is a human resource company that makes work better through science and machine learning. Rippling Rippling provides cloud-based employee management solutions, from HR (payroll, benefits, compliance) to IT (computers, software, security). Truework Truework is an HR tool that automatically responds to lenders, landlords, and others who need to verify employment. Seize any opportunity, or anything that looks like opportunity. They are rare, much rarer than you think. — Taleb Healthcare / Biology Basis Basis is an app that allows people to deal and cope with mental health issues and offers same-day access to trained specialists. CoFactor Genomics The Leaders in RNA. Harness the power of multidimensional gene expression models to treat cancer more effectively. Forward Forward provides you with personalized primary care, combining world-class doctors with advanced medical technology including genetics, cancer screening & wellness tracking. Gyroscope Gyroscope is a new OS for the human body, with an app and website for people to view their health analytics. Hims Hims is an e-commerce wellness brand made for men with products for skin care, erectile dysfunction, and hair loss. They work with doctors to prescribe Finasteride and Sildenafil (active ingredient in Viagra) through the app or website. MDAcne MDacne is a mobile platform that uses AI and computer vision to help people with acne. Ro Ro is a mission-driven healthcare technology company that handles everything from online diagnosis to the delivery of medication. Synthego Founded by former SpaceX engineers, Synthego is a leading provider of genome engineering solutions. Their flagship product, CRISPRevolution, is a portfolio of synthetic RNA designed for CRISPR genome editing and research. Zipline Zipline builds products that improve access to healthcare and saves lives. I always did something I was a little not ready to do. I think that’s how you grow. When there’s that moment of ‘Wow, I’m not really sure I can do this,’ and you push through those moments, that’s when you have a breakthrough. — Marissa Meyer Food Cloud Kitchens Powering the on-demand food economy. They provide infrastructure and software that enables food operators to open delivery-only locations with minimal capital expenditure and time. Full Harvest Full Harvest helps growers get the most out of a harvest and food companies save money by bringing farm excess to market. They are a B2B platform connecting large farms to food businesses to sell significantly discounted, yet perfectly good surplus and imperfectly shaped produce that would have otherwise gone to waste. Impossible Impossible Foods develops a new generation of meats and cheeses made entirely from plants. They select specific proteins and nutrients from greens, seeds, and grains to recreate the experience of meats and dairy products. Uber Eats (disclaimer: I interned at Uber) Uber Eats is a food delivery platform that makes getting great food from a favorite local restaurants as easy as requesting a ride. Sensei SENSEI is here to guide you towards greater wellbeing. Their ecosystem of products and experiences is designed with one intention: to empower you to grow well. “Two ways to be successful in life: do something worth writing about or write something worth reading.” — Ben Franklin Government Anduril Anduril commits top technical talent to solving the most complex national security challenges. They’re building the next generation of technology that will aid and protect those who serve on the front lines defending the nation and its interests. Citizen The need has never been greater for technology that informs and protects the public. The Citizen app empowers people with information to help keep their families, communities, and cities safe. Users receive instant notifications every time a crime or other major incident is reported near them. Coord Coord is a mobility company that creates seamless mobility and self-driving experiences today through deep integrations. They offer bike-share API, Curbs API, Tolls API, Routing API and etc. Sidewalk Labs Sidewalk Labs is a new type of company that works with cities to build products addressing big urban problems. They’re building a platform and a set of urban applications to accelerate innovation in cities around the world. Whenever you find a barrel , you should hire them instantly, regardless of whether you have money for them or whether you have a role for them. Just close them. — Keith Rabois Logistics Convoy Convoy is a network of tech-enabled trucking companies transforming the $800B trucking industry and logistics ecosystem for truck drivers and shippers. Using technology, Convoy matches reliable trucks (“carriers”) with companies that need to ship freight. Flexport At Flexport, they believe that a world that does business together is a better one. It’s called global trade. And freight forwarding is the $1 trillion industry that functions as the circulatory system of global trade — without freight forwarding, global trade simply wouldn’t exist. Nabis Nabis delivers the most reliable, consistent, and secure logistics solution for cannabis businesses. Whether you are a brand, cultivator, intermediary processor, or final good manufacturer, Nabis provides you with access to an enterprise-scale distribution network. Nuro NURO is a technology company which aims to accelerate the benefits of robotics for everyday life. They are a team of entrepreneurs and engineers, designers, and scientists who believe the advancements they are making in robotics and machine learning today will dramatically improve the way they live tomorrow. Rubicon Rubicon Global is a technology company that provides waste, recycling, and smart city solutions to businesses and governments worldwide. They develop and deploys a full suite of cloud-based solutions to help its customers and network of independent waste haulers work towards more sustainable waste and recycling practices, uncover cost savings, and improve operations through data analytics. Uber for Business (disclaimer: I interned at Uber) Get an all-in-one solution that simplifies how businesses gets around. It’s a digital “headquarters” for all your company’s ground transportation. Get a clear view into all your trip activity and automate billing, expensing, and reporting. Easily provide rides — even multiple rides at the same time — to anyone important to your business. Uber Freight (disclaimer: I interned at Uber) Uber Freight is an app that matches carriers with shippers. Carriers can just tap a button and instantly book a load, and the price you see is the price you get — no more back-and-forth negotiations. Make friends over the internet with people who are great at things you’re interested in. The internet is one of the biggest advantages you have over prior generations. Leverage it. — Patrick Collison Real Estate / Construction Buildzoom BuildZoom has a simple purpose: to help you find the perfect contractor for any job — commercial or residential. Using BuildZoom is just like asking a neighbor for a recommendation — except instead of just asking one, you are polling every neighbor in your community, analyzing their feedback, exhaustively researching work histories and credentials and only then, making a final decision. Bungalow Bungalow is a residential real estate platform that provides renters with a more convenient, flexible, and communal living solution. It utilizes existing housing supply by signing long-term leases with homeowners, and offers multi-bedroom homes in some of the most desirable neighborhoods in cities throughout the United States. Opendoor Opendoor makes it possible to receive an offer on a home in just a few clicks, and sell in a matter of days, removing all of the headaches, uncertainties, and risks from the transaction. They are headquartered in San Francisco, California and has received a total of $645 million in venture funding from Khosla Ventures, GGV Capital, and Access… Qualia Qualia is a cloud-based closing platform that provides seamless and secure coordination between everyone involved in the real estate closing process. Wanderjaunt WanderJaunt offers technology-enabled property management solutions for travelers, homeowners, and accommodations from around the world. And then the last piece is actually the hardest, which is specific knowledge. Specific knowledge is knowledge that you have that lets you do something that other people do not know how to do and cannot be trained to do. — Naval Ravikant Education / Life / Talent Holloway With hundreds of hours of research, writing, and editing, their comprehensive Guides help you make better decisions. Hatch Baby Their mission is to bring parents a series of smart products that use technology and user-centric design to help make caring for a baby a little easier. Lambda School Lambda School trains people to be software engineers for free in exchange for a share of future income. Pioneer A home for the ambitious outsiders of the world. They’re building a community of creative young people working on interesting projects around the globe. TripleByte Triplebyte believes the current technical hiring process doesn’t do enough to help engineers show their strengths. They’re dedicated to building a better process. You can read more about their views on hiring in their manifesto. 70MillionJobs 70MillionJobs is the first for-profit employment job board for people with criminal records. “Everyone starts out with very, very humble beginnings” — Daniel Gross Social / Consumer Brandless Brandless is an e-commerce company providing unbranded and essential products. Jumprope Create and share recipe how-to’s. Export to Instagram Stories, IGTV, YouTube, your personal website and more. Masterclass MasterClass is a premier online education platform redefining the digital learning experience by bringing world-class experts and renowned instructors to students globally through a cutting-edge technology platform meant to distill knowledge and create meaningful experiences through online video. Sandbox VR Sandbox VR combines the latest VR hardware, movie quality real-time motion capture and AAA-quality content experiences to create the best VR experiences on the planet. Period. It’s the perfect event for team building, corporate events, and parties. People always say you should live your life as if it were your last day. I think you should live your life as though it will go on for ever; that every day is so good that you don’t want it to end. — Peter Thiel Dev Augur Augur is built on Ethereum, a decentralized blockchain network similar to the Bitcoin network. Utilizing this technology, their markets can effectively eliminate counterparty risk and centralized servers while creating a global market by employing cryptocurrencies such as bitcoin. Clearbit Clearbit develops business Intelligence APIs. Clearbit is on a mission to set the new standard in business data. Simply put — Clearbit is changing how modern businesses source and operationalize data as part of their sales, marketing, and product workflows. Deepgram Deepgram saves businesses time and money by building tailored speech recognition models that increase revenue and maximize productivity. Handshake Decentralized certificate authority and naming. VectorDash Vectordash lets you use insanely fast cloud GPUs at incredibly low costs. You can start a powerful GPU instance within seconds. The world is an incredibly complex place and everything is changing all the time. You can’t plan your career because you have no idea what’s going to happen in the future. You have no idea what industries you’ll enter, what companies you’ll work for, what roles you’ll have, where you’ll live, or what you will ultimately contribute to the world. You’ll change, industries will change, the world will change, and you can’t possibly predict any of it. — Marc Andreessen SaaS Tools Density Density is a modern infrastructure for anonymously counting people. Front Front redefines work communication with the first shared inbox for teams. By unifying your email, customer communication channels, and apps in one platform, Front helps teams collaborate efficiently and have more context and visibility into every conversation, to work faster and better together. Fullstory FullStory takes a fundamentally different approach to understanding your customer experience. One tiny script unlocks pixel-perfect playback, automatic insights, funnel analytics, and robust search and segmentation — all without the need to manually tag your site’s code. Openphone OpenPhone adds a second phone number on top of one’s existing mobile device with powerful features to help business stand out. Mixmax Mixmax offers software that enables its users to boost their productivity. It enables its users to track emails accurately, set up meetings in an instant, save time with email templates, and schedule emails to be sent later. Notion All-in-one workspace. Write, plan, collaborate, and get organized. Notion is all you need — in one tool. Superhuman Superhuman is not just another email client. They are rebuilding the inbox from the ground up to make you brilliant at what you do. They specifically designed it for those of you who want the best. Miscellaneous 100 Thieves 368 Aero Angel List Apple Health Atoms b8ta Culdesac Deos Houseparty Neon Panda Saildrone Social Construct Substack Transmission Ttyl Urbit have thoughts? need help? i'd love to chat :)

message me on twitter or here - jordan gonen 🌊 Hope this was valuable! If you enjoyed, please share and hit recommend 💙 Company descriptions are from . Crunchbase.com Thanks Jonathan Z. White & David Lee for feedback/edits.

The Signal

Photo by on Rodion Kutsaev Unsplash Quantopian — The Online Algo Trading Platform is one of the most popular online algo trading platforms and communities today. It provides the great backtesting environment where you can experiment with your idea, build and even participate in the contest, as well as share the idea and discuss it with smart people there. Quantopian algorithms One of the things many people have asked Alpaca during the beta program is how to run the algorithms that they built in Quantopian platform for their own purpose, not just for the contest. While Quantopian has built so much in the platform, they are so great to share the internal framework as open source zipline. The Newest Open Source Libraries for Quantopian Users Today, I wanted to share our newest open source libraries for Quantopian users; and . pylivetrader pipeline-live _Python live trade execution library with zipline interface. - alpacahq/pylivetrader_github.com alpacahq/pylivetrader _Pipeline Extension for Live Trading. Contribute to alpacahq/pipeline-live development by creating an account on GitHub._github.com alpacahq/pipeline-live pylivetrader is a zipline API compatible trading framework in which again focuses on live trading, with much less overhead and dependency problems. It is written from the ground up for live trading use cases, so it removes a lot of heavy lifting that zipline had to do such as price adjustment etc. python This means, you don’t need to build your data bundle to kick off your algorithm in live, but instead you can just start your live trading from the Quantopian algorithm source right away. At the moment, the supported backend is only , but we are happy to connect to IB etc. if someone contributes the code. Alpaca Pipeline API — the Core Piece of Quantopian Framework Pipeline API is the core piece of Quantopian algorithm framework that allows easy stock selection based on the different metrics, much in a pythonic way, and this differentiates the platform from others. I found Pipeline is providing a tremendous value when it comes to trading wide range of universe. Unfortunately, it is not so easy for most people to use this great feature outside of the Quantopian platform. is a python tool that allows you to do something similar anywhere so that you can do your research somewhere else as well as use it with existing python trading framework such as zipline-live or backtrader, including pylivetrader which I am introducing below. pipeline-live primarily uses IEX public API for pricing and basic fundamental information. pipeline-live As you know, IEX provides market-wide volume data for daily OHLCV which makes it a perfect choice for pipeline usage. Since pipeline-live focuses on live trading use cases, it does not provide historical view unlike inside Quantopian, but the upside is it is fairly independent and easy to use. It is also very extensible so you can hook up with other paid data sources if you would find useful. How to Convert Your Quantopian Algorithms to Run in Live Trading We also put some practices together about how you could convert your Quantopian algorithms to run in live trading. You may want to take a look at these documents if you are interested in. https://github.com/alpacahq/pipeline-live/blob/master/migration.md https://github.com/alpacahq/pylivetrader/blob/master/migration.md I also posted in Quantopian forum with the real example, and you may take a look at it, too. _This is a modified version of the algorithm presented in…_www.quantopian.com Long-only non-day trading algorithm for live Feel free to give me any feedback/questions/criticism. Happy to help you get started with live trading with these tools too. Below is the example code migrated from the post above. Please follow and for fresh posts on Financial Market, Algorithmic Trading, Technology. Alpaca Automation Generation You can find us , if you use twitter. @AlpacaHQ If you’re a hacker and can create something cool that works in the financial market, please check out our project “ Commission Free Stock Trading API ” where we provide simple REST Trading API and real-time market data for free. Brokerage services are provided by Alpaca Securities LLC ( ), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc. alpaca.markets

Python Library To Run Quantopian Algorithm In Live

By Chris Lee on Unsplash Hey everyone, Intern here. A few weeks ago, I worked with and to put together “9 Great Tools for Algo Trading”. Rao Hitoshi Yoshi 9 Great Tools for Algo Trading _In the last 5–10 years algorithmic trading, or algo trading, has gained popularity with the individual investor. The…_hackernoon.com Looking at the response to that article, we decided to write a follow up. Whether you’re a financial firm or an individual trader, financial is key for putting together any good strategy. With so many on the market today, many good options get lost in the noise. Here are 11 great financial data vendors. data vendors If your service is featured and is being misrepresented in some form, please reach out to me ASAP (through my medium page, linked at the bottom), and an immediate correction will be made. Vendors for the Individual Investor The following data vendors are those that see their target audience as the individual investor. While they do have larger premium plans for high volume usage, the various user plans and community are structured around the individual investor. (1) Nasdaq Nasdaq Inc, founded in 1971, owns and operates the NASDAQ — currently the second largest exchange in the world (according to market cap). They also acquired several European exchanges after their acquisition of OMX, the former Swedish-Finnish company that owned several Nordic and Baltic exchanges. Nasdaq offers free historical data for personal use. Users have the ability to look up a max of 25 symbols at a time, and then download up to 10 years of historical EOD data in .csv or excel format. Nasdaq also has several products live data streams. These products include data US Equities traded on the NASDAQ as well as the eight foreign exchanges they own. While NASDAQ’s pricier products can run as high as $1200 a month, it’s free historical data, lookup as well as basic API means that it’s within the means for the individual trader. (2) : CME Group The CME Group is a financial market company operating out of Chicago. They formed from the 2007 merger of the Chicago Market Exchange and Since 2010, the CME Group has expanded by buying ownership shares in the S&P Dow Jones Indices, the Kansas City Board of Trade, and the British NEX Group. Currently, the CME Group offers both historical market data, and live data planes. Historical market data is sold for a one-time payment, and is priced based on depth and scope of the data provided. The CME Group also provides the Market Data Platform, which disseminates real time market data. (3) Quandl If you’ve used Zipline for backtesting at some point, you’ve probably used free data from Quandl. Based out of Toronto, Quandl first started as a search engine for financial data sets. As algo trading grew as a field, Quandl started repackaging their search engine of data sets as individual bundles. Quandl offers a bundle with US Equities for free, which is actually the default bundle you ingest on Quantopian’s Zipline backtesting engine. Keep in mind that this bundle has EOD data extending to February 28th, 2018 and only contains US Equities. Other bundles with larger scopes and up-to-date data, are priced for a one time fee. Quandl even serves as a market-place for third parties to sell their bundles. If you’re newer to the field, Quandl’s not a bad place to start. (4) Alpha Vantage AlphaVantage is a community-based data vendor. In the last few years, AlphaVantage have worked hard to foster a developer community, not unlike Quantopian. The community helped build their product, as well as discuss investment strategies and other apps built on AlphaVantage data. AlphaVantage is a generally free resource with APIs available in multiple languages. They provide both real-time data streams and historical market data for backtesting. The API doesn’t have a daily/weekly/monthly limit, but the free subscription does have limited latency. For the serious investor with larger API calls, a premium subscription is best. (5) Intrinio Intrinio was founded in 2012, with the idea of making financial data easy to access. Intrinio actually gears their product towards developers, and is currently the data provider for the backtesting engine QuantRocket. QuantRocket - Automated International Trading with Python and Interactive Brokers _Yet working with IB data can be challenging: incomplete documentation, pacing violations, an event-driven programming…_www.quantrocket.com They currently offer two forms of data — streams and packets. Data streams offer a free trial, and then are subscription based depending on the stream. Packets are bundles of historical data. These bundles can be bought for a one-time payment and are priced based on the width and depth of data coverage. They also offer applications like the Intrinio screener for Excel. (6) Zacks Zacks was founded in 1985 as a quantitative ranking of potential buys based on estimated earnings. That tool Zacks Ranks was the cornerstone of what Zacks would come to offer today. Zacks is more than just a data vendor. It offers both historical and real-time data, as well as a variety of third party applications to use that data. Zacks also features newsletters, advice from industry experts, and a community of developers. For traders who are looking to wet their feet, Zacks may be the place to go. (7) Polygon Polygon is far newer on the data vendor scene, having been founded in 2015. They have a variety of pricing plans based on data coverage as well as latency of connection. Polygon provides real-time quotes and data streams, and have official clients in 10 different languages including Python, C, and Go. In a trend keeping with other data vendors of late, Polygon has done their best to foster a community of developers, which has led to the creation of community clients including Perl, .NET, and Scala. Disclaimer: Polygon is the data vendor of choice for Alpaca (8) EOD Historical Data Stock Price Data, Financial and Stock Market API, Real-Time Currency and Crypto _Cheapest stock historical data and prices on the market. Best End-of-Day Data. More than 30+ stock exchanges and…_eodhistoricaldata.com Quirkily named EOD Historical Data was founded in May of last Year, but has established itself with a combination of accurate data and affordable prices. The Lyon, France based company offers monthly rates that range from as low as $9.99 to a high of $39.99. (9) IEX IEX, which stands for the Investor’s Exchange, was founded by four former employees of the Royal Bank of Canada. Upset by the unfair conditions of today’s stock market, they looked to create an exchange that leveled the playing field for long-term investors. Their story was chronicled in the 2014 Michael Lewis Book . Flash Boys: A Wall Street Revolt Flash Boys: A Wall Street Revolt _1 New York Times Bestseller ― With a new Afterword”Guaranteed to make blood boil.” ―Janet Maslin, New York TimesIn…_www.amazon.com IEX has a significant amount of free data, as well as multiple APIs to access data. The catch is though that all pricing data is the price of that security on IEX. That said, their coverage of securities is second-to-none, as they have information on US equities, fixed income, and ETFs. Find out more on the IEX website . here Business-Facing Vendors The vendors listed in the previous section are geared towards the individual investor, they have premium subscriptions for businesses. These following vendors are specifically geared towards firms, and are out of the price range of the average individual investor. (10) Xignite Xignite was initially founded as a wealth management platform. When the founder Stephane Dubois realized the challenge of obtaining accurate market data, he pivoted Xignite to be come a data vendor. Xignite offers real time data streams, daily quotes, and historical market data. For developers, they offer multiple APIs for historical data. Their products are expensive for the individual investor, and Xignite themselves identify as business facing. Some of their more famous clients include Robinhood, Wealthfront, and StockTwits. (11) Thomson Reuters Thomson Reuters is the merger of mass multimedia Thomson Company and the information group Reuters. Founded in London, Reuters has been providing financial information since 1951. Like Bloomberg, Reuters transitioned into information providing in the late 20th century. Reuter’s provides pricing data for assets in over 200 exchanges for both equities and fixed income. Their software includes not just pricing data, but research and analytical tools, along with a dedicated news stream, and a mobile interface. The full Eikon software costs $22,000 a year, but individual investors can find a stripped down version with just pricing data for $3,600. (12) Bloomberg Bloomberg was founded in 1981 by former New York City mayor Michael Bloomberg. Bloomberg may not have Reuter’s pedigree, but it’s quickly established itself as industry standard. So industry standard that it made an appearance in Season 3 of . The Newsroom Scene taken from Season 3 Episode 01 of the Newsroom on HBO The Bloomberg Terminal packs minute by minute data, trading tools and analytics for over 300 exchanges around the world. For algo traders, the terminal subscription even offers an API to access in multiple languages. But clocking in at $24,000 a year, it’s a hard sell for the individual investor. If you have access, whether through your firm or institution, you can’t get better than Bloomberg. do (13) YCharts YCharts was founded by Ara Anjargolian and Shawn Carpenter as an attempt to compete directly with the Bloomberg Terminal. Founded in 2009, the self proclaimed financial terminal of the web balances a load of data along with a variety of useful tools and integration. They offer Excel integrations and data visualizations of niche metrics, which is in line with their target audience; hedge fund managers and sales representatives. Both groups are accustomed to in-depth research on behalf of their clients. YCharts Professional clocks in at $199/month which is expensive, but only about 10% of the cost of a Bloomberg Terminal. For those interested in a stripped down subscription with only access to the simplest data metrics, YCharts offers a membership at $49/month, which is affordable for most individual investors. I hope this data vendor list was useful. If you liked it, please leave a clap (or fifty, I don’t mind). If you think there are services that I missed leave a comment below! I always appreciate any, and all feedback. by Rao Vinnakota Edit (08/03/18): - Information about Quandl’s data offering was outdated. They no longer offer free EOD Data past February 28th. - CME and Nasdaq were added as Data Vendors, changing the title from 11 to 13. - The list was also re-ordered. Please follow and for fresh posts on Financial Market, Algorithmic Trading, Technology. Alpaca Automation Generation You can find us , if you use twitter. @AlpacaHQ If you’re a hacker and can create something cool that works in the financial market, please check out our project “ ” where we provide simple REST Trading API and real-time market data for free. Commission Free Stock Trading API Brokerage services are provided by Alpaca Securities LLC ( ), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc. alpaca.markets

Data, Data, Data! 11(+2) Great Financial Data Vendors

Photo by on Adrian Curiel Unsplash In the last 5–10 years algorithmic trading, or , has gained popularity with the individual investor. The rise in popularity has been accompanied by a proliferation of tools and services, to both test and trade with algorithms. I’ve put together a list of 9 tools you should consider using for your algo trading process. algo trading Web Services: The following are managed-services that you can use through web browsers, and don’t require much setup from the user. As someone who’s recently started in this field, I found it easy for new algo traders to try out. (1) : Quantopian A Boston-based crowd-sourced hedge fund, Quantopian provides an online IDE to backtest algorithms. Their platform is built with python, and all algorithms are implemented in Python. When testing algorithms, users have the option of a quick backtest, or a larger full backtest, and are provided the visual of portfolio performance. Live-trading was discontinued in September 2017, but still provide a large range of historical data. They also have a serious community of developers, and post an ongoing DAILY contest with 10 winners awarded each day for a total of $5000 per month in prize money (*updated from the previously written as “periodically hold contests”). Quantopian provides capital to the winning algorithm. _Algorithm writers win thousands of dollars each month in this quant finance contest. Quantopian provides the education…_www.quantopian.com Quantopian Contest (2) : QuantConnect QuantConnect, is another platform that provides an IDE to both backtest and live-trade algorithmically. Their platform was built using C#, and users have the options to test algorithms in multiple languages, including both C# and Python. QuantConnect also embraces a great community from all over the world, and provides access to equities, futures, forex and . They offer live-trading integration with various names such as InteractiveBrokers, OANDA, and GDAX. crypto trading (3) QuantRocket: QuantRocket is a platform that offers both backtesting and live trading with InteractiveBrokers, with live trading capabilities on forex as well as US equities. It’s specifically designed for trading with InteractiveBrokers, and sets itself apart with its flexibility. QuantRocket supports multiple engines — its own Moonshot, as well as third party engines chosen by the user. While QuantRocket doesn’t have a traditional IDE, it is integrated well with Jupyter to produce something similar. One thing to keep in mind is that QuantRocket . Pricing plans start at 19.99/month USD, with annual options. is not free Local Backtesting/LiveTrading Engines: In today’s software world, you have lots more freedom if you make some effort outside of those managed-services. If you are comfortable this way, I recommend backtesting locally with these tools: (4) : Zipline/Zipline-Live _zipline - Zipline, a Pythonic Algorithmic Trading Library_github.com quantopian/zipline Quantopian’s IDE is built on the back of Zipline, an open source backtesting engine for trading algorithms. Zipline runs locally, and can be configured to run in virtual environments and Docker containers as well. Zipline comes with all of Quantopian’s functions, but not all of its data. To balance that, users can write custom data to backtest on. Zipline also provides raw data from backtests, allowing for versatile uses of visualization. Zipline discontinued live trading in 2017, but there is an open source project that works with Interactive Brokers. It has many of the same features Zipline does, and provides live trading. Zipline-live (5) : BackTrader _from datetime import datetime import backtrader as bt class SmaCross(bt.SignalStrategy): params = (('pfast', 10)…_www.backtrader.com backtrader - Backtesting / Trading Backtrader is currently one of the most popular backtesting engines available. It was built using python, and has a clean, simple, and efficient interface that runs locally (no Web Interface). One thing to keep in mind, backtrader doesn’t come with any data, but you can hook up your own market data in csv and other formats pretty easily. Starting with release 1.5.0, BackTrader has live-trading capabilities. It’s been a popular choice with algo traders, especially after Zipline discontinued live trading. (6) : IBPy _IbPy - Python API for the Interactive Brokers on-line trading system._github.com blampe/IbPy IBPy is an unaffiliated third party python wrapper for InteractiveBroker’s Trade Workstation API. Before IB started providing their official API library for python, this was the only way to connect to TWS for algorithms written in python. IB has released an official python SDK, and this library is heading towards begin obsolete(while still being relevant for python2 users). But there still remain a significant number of live trading engines/tools that still use this library, and it’s good learning material for whoever wants to learn about implementing API’s. While it’s good to learn about this library since it’s ubiquitous, if you are starting fresh, we recommend IB’s official python SDK. is even much simpler to use! Alpaca Trade API Python SDK Analytical Tools: Back testing will output a significant amount of raw data. Some IDE’s will provide basic visualization and analysis, usually algorithm performance. If you’re looking for deeper evaluation, I recommend these tools: (7) : Pyfolio _pyfolio - Portfolio and risk analytics in Python_github.com quantopian/pyfolio Pyfolio is another open source tool developed by Quantopian that focuses on evaluating a portfolio. What sets Pyfolio apart, is its ability to introduce degrees of uncertainty to a static set of data points, and evaluate Bayesian metrics from the user’s portfolio. The Pyfolio API offers a number of visualizations, which can be found on their GitHub repository. (8) : Alphalens _alphalens - Performance analysis of predictive (alpha) stock factors_github.com quantopian/alphalens Alphalens is also an analysis tool from Quantopian. Unlike Pyfolio, Alphalens works well with the raw data output from Zipline, and rather than evaluate the portfolio, is performance analysis of predictive stock factors. Alphalens has its own range of visualizations found on their GitHub repository. Median Daily Returns by Factor Quantile — one of the visualizations that alphalens offers (9) : TradingView TradingView is a visualization tool with a vibrant open-source community. It’s entirely web-based, and allows users to visualize data, whether the data is the result of paper trading or algorithmic back-testing. Like Quantopian, TradingView allows users to share their results and visualizations with others in the community, and receive feedback. (Bonus) Execution Platforms aka Broker-Dealers: (10)InteractiveBrokers: InteractiveBrokers is an online broker-dealer for active traders in general. They have been in the market since 1978. Algo trading isn’t IB’s focus, but multiple engines offer live trading through integration with their Trader Workstation. We’ve mentioned IB several times in this article — they’re just that good! (11)Alpaca: Finally, Alpaca! Alpaca was founded in 2015, and is an up and coming commission-free, broker-dealer designed specifically for algo trading. Alpaca also has a trade api, along with multiple open-source tools, which include a database optimized for time-series financial data known as the . MarketStore The brokerage is scheduled to be publicly available this September (you can play around with the MarketStore right now), but if you can’t wait, head over to our website and for a chance at early access! jump on the waitlist _Lower your cost, Maximize your profits. Algo trading commision free._alpaca.markets Alpaca | Algo Trading Commission Free with REST API Miscellaneous Tools to Take a Look At: — another simplistic python backtesting engine qtpylib — proprietary trading platform for forex and equities Multicharts — desktop tool which allows C# backtesting, with live trading exclusive to Fidelity WealthLab — for crypto trading Enygma Catalyst — backtesting/livetrading desktop app, de-fact in forex MetaTrader I hope this quick primer on tools available right now was useful. If you liked it, please leave a clap (or two, I don’t mind). If you think there are tools that I missed, leave a comment below! I always appreciate any, and all feedback. by Rao Vinnakota Please follow Alpaca and Automation Generation for fresh posts on Financial Market, Algorithmic Trading, Technology. You can find us @AlpacaHQ , if you use twitter. If you’re a hacker and can create something cool that works in the financial market, please check out our project “ Commission Free Stock Trading API ” where we provide simple REST Trading API and real-time market data for free. Brokerage services are provided by Alpaca Securities LLC ( ), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc. alpaca.markets

9 Great Tools for Algo Trading

All code and outside instructions for setup, etc. can be found in the GitHub repository here. This is a post is a sequel to building a robo advisor in Quantopian. You can find the first part on Hackernoon here , or on Alpaca’s blog here . _I'm Rao , and I'm an intern at Alpaca working on building an open-source robo advisor. I don't have much experience in…_blog.alpaca.markets I am a College Student and I Built My Own Robo Advisor Hey all, I’m Rao, an intern at Alpaca. About two weeks ago, I published an article on HackerNoon about the Robo-Advisor I’ve been working on. In the last two weeks, I’ve moved my work offline, pushing development with Zipline. In that time, I’ve picked up a few tips and tricks, and I wanted to share that with you. I’m…still…working… Zipline — What, When, Why, Where, How? We’ve now created a functional robo advisor on Quantopian. But, the IDE puts limits on the scale and sophistication of our work. To extend this robo advisor further, it’s best to do our future work locally with Zipline. This shift offers advantages. We can use multiple files, configuration files, and generally develop out robo-advisor like any other python app. Being free from a single file format means that our work can be more organized, and individual functions less bloated (more on that later). But, it’s not all roses. Developing in Zipline comes with its own challenges. But, if I can do it, you can too! Installing zipline on your local machine isn’t as simple as `pip install zipline`. It has several dependencies, which vary slightly for each OS. You can find all necessary installation instructions . here I made several mistakes during installation, so I recommend you run zipline in either a virtual environment, or my personal preference, in a Docker container. I’ve attached both the finished container to play with, as well as the corresponding Dockerfile. Take some time, figure out your development environment. Ready? Let’s get started. Zipline Commands — Bundles and Execution Open a new python file locally. Copy and paste your code from the Quantopian IDE. Now, we’re ready to go- just yanking you around. We’re not close to ready. Quantopian’s IDE is a great development environment. It takes care of background processes, allowing you to focus solely on your algorithm. First, we need historical data to run our algorithm on. Zipline reads through data using something called bundles. To use the bundles, they need to be ingested first, which is done using the command: $zipline ingest -b <bundle> Zipline provides two bundles, quandl and quantopian-quandl. To ingest quandl, you’ll need to make an account, and then obtain an API key. You won’t need to for quantopian-quandl. Zipline also provides the option to create custom bundles, by writing a custom ingest function. That will be reserved for a later post. Both Quandl and quantopian-quandl don’t provide ETF data, so I’ve provided a custom alpaca bundle. The instructions for installing this bundle locally can be found in the GitHub repository readme. You run zipline code using the following format: $zipline run -f <filename> -b <bundle> --start <date> --end <date> You can see a lot of the GUI input from Quantopian is configured from the command line with Zipline. Docker Containers: I’ve created a docker container running python 3.6 with zipline and necessary dependencies installed. If you’re hesitant about installing zipline locally, you can pull the container from the , and experiment with the environment. docker hub Run the image with the command: $docker run -it alpaca/roboadvisor /bin/bash A First Zipline Example — Buy and Hold Like my initial start with Quantopian, I started with Zipline using a simple buy and hold strategy. Here are a few differences to keep in mind. First, all the custom functions like order, symbols, etc. are no longer automatically included, and have to be manually imported ( ). Next, this is rather silly, but now that we’re not in Quantopian’s IDE, we’re no longer going to use log.info to track transactions, but print statements. from zipline.api import * Let’s run the code with the format from earlier, and run the tests from January to June: $zipline run -f buy-and-hold.py -b alpaca --start 2018-01-01 --end 2018-06-01 If there’re no syntax errors, Zipline will spit out a whole bunch of data. How do we know if we’re right? Scroll until you find the following table from STDOUT: Look at the cumulative alpha value from the last simulated trading day (2018–06–01) Find the cumulative alpha value (performance against benchmark), and compare it to the cumulative alpha value when you run the same algorithm on Quantopian. They should be very similar values. We’ve got a good idea of how Zipline works, so we can go ahead and implement the rest of our single-universe algorithm (distance computation and rebalancing). I spent an entire post talking about this implementation, so I don’t think there’s a need to re-hash it. If you haven’t read that, I recommend you do — it’s a great read! (a totally unbiased opinion) Multiple Universes In the previous post, I showed how to expand the algorithm to cover all possible Vanguard universes by adding multiple dictionaries in initialize function. But, if we want to implement all six vanguard universes, we would finish with a bloated ingest function, which doesn’t look that pretty. But now we’re away from Quantopian, and here with Zipline! We can spread our code across multiple files, so let’s go ahead and do that. To be used by the algorithm, a universe needs two sets of information. The first is the set of symbols, and the second is the weight distribution based on the risk. From the way that the symbols function works, each universe’s list of symbols needs to set in the initialization function. But, the weights are dictionaries that aren’t bound to any zipline.api functions, so we can actually configure those in a separate file. The ConfigParser package can read in data stored in an external .ini file. More importantly, it stores the data in a similar structure to a dictionary (key/value), and is easily called and organized. First, we need to install the package: $pip3 install ConfigParser With the package installed, it’s time to create the INI file. The INI file contains the information for allocation based on risk level. INI files are separated into sections, with a new section delineated by a section title in brackets. The section titles act much like a key in dictionaries, and is a useful way to organize inputs. This file organized each individual Vanguard universe as its own section. If you’d like to add your own universe, fork the gist, create a new section with the universe name in brackets, and list the information below it. The robo-advisor’s algorithm parses input as a dictionary. While the INI file works much like a dictionary, we’ll still need to write a function to actually translate it into a dictionary. def section_to_dict(section):config = ConfigParser()config.read('universe-config.ini')out_dict = {}for key in config[section]:out_dict[int(key)] = ast.literal_eval(config[section][key])return(out_dict) The configparser is initialized, and then reads the given INI file. The section that’s read is given as user input. If we refer back to the INI file, we’ll see that the values take the form as key = value. So like a dictionary, we can iterate through the deys of an INI section. In each case, each key/value pair is added to the dictionary. ( Note: If you receive a key error, it’s because the path to your config.ini file is wrong Two more things to be aware of. In the INI file, everything is a string, but our algorithm expects an integer/tuple key-value pairs. For the key, just casting them as integers is enough, but for the tuple, we’ll have to unstring the value. For that, I found ast.literal_evalto be the function that worked best. Now, let’s integrate this new method of retrieving weight-based allocation with our robo-advisor algorithm: I’ve listed all the symbols for every universe in the initialization. The weights for that universe are determined by calling section_to_dict on the appropriate section of the INI file. Now, feel free to add as many universes as you like! What’s Next? Going from here, I’m interested in visualizing all the output data from the Zipline backtest. With my next post, I’m hoping to explore the different visualizations available from that raw data. by Rao Vinnakota Please follow and for fresh posts on Financial Market, Algorithmic Trading, Technology. Alpaca Automation Generation You can find us , if you use twitter. @AlpacaHQ If you’re a hacker and can create something cool that works in the financial market, please check out our project “ Commission Free Stock Trading API ” where we provide simple REST Trading API and real-time market data for free. Brokerage services are provided by Alpaca Securities LLC ( ), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc. alpaca.markets A special thanks to Hitoshi, Yoshi, and the rest of the Alpaca team for all the help and tech support while I figured out Zipline and Docker.

Stories, News & Mentions about ZIPLINE

Zipline’s Stories on HackerNoon