Best match

Most recent

Strike

A total of 1,650 North Sea oil and gas workers will walk out today, Thursday 1 June 2023, as part of the sector’s biggest industrial action over profiteering, pay and conditions. Rosemary Harris will be introducing a showing of the Offshore documentary about energy workers at the Cinema Climatic film showing at the Paintworks in Bristol on Saturday,

https://www.bing.com/th?id=OVFT.kV8HeP_qr-5kziVdOLM9ri&pid=News

Oil and gas workers set to strike

https://www.bing.com/th?id=ODF.vnmEU3x5Er-iGBstczl31g&pid=news

The Ecologist

Racing

TVS has a well-rounded portfolio in Colombia, spanning various categories like commuter bikes, the Apache bikes, and scooters like the Ntorq

https://www.bing.com/th?id=OVFT.z93xMYnzTyxdfS9NFXugny&pid=News

TVS Apache 200 Racing Edition – New Colour, Improved Safety

https://www.bing.com/th?id=ODF.vZE3MsYdzw5rpc5h3nxULw&pid=news

Rushlane.com

Colombia

Special edition

Apart from the 125cc premium commuter, TVS Motor Company has also introduced the Apache RTR 200 4V Special Edition Racing in Colombia.

https://www.bing.com/th?id=OVFT.XMbj71MoAky8j2Hy-7p8dS&pid=News

TVS Apache

TVS Apache Special Edition Racing variant launched in Colombia

https://www.bing.com/th?id=ODF.y4FzXrtX5pHWN1hg6XjA4w&pid=news

BikeWale

The Karankawa peoples in South Texas were said to be extinct. But descendants today are reclaiming and fighting to protect their heritage.

Reclaiming heritage, some in the Coastal Bend fight to preserve Karankawa land, history

https://www.bing.com/th?id=ODF.2T6Kzu0T0crPSCqfi-x_dQ&pid=news

Corpus Christi Caller-Times

Suriname

Suriname’s long-awaited oil boom could finally get underway with a growing number of indicators that a final investment decision on Block 58 could be made this year.

https://www.bing.com/th?id=OVFT.9uHl3pKHeEC8XDL7sfeQDy&pid=News

Suriname’s Much-Needed Oil Boom May Soon Be Back On Course

https://www.bing.com/th?id=ODF.jE5hGDMmxgr3hnRLnOXIpw&pid=news

Crude Oil Prices

Guyana

Momentum

Guyana’s oil boom continues to gather momentum, and with new discoveries being made, the small South American country is poised for very fast economic growth

https://www.bing.com/th?id=OVFT.DYScP7hw3TF7q27DSIzL1y&pid=News

Guyana's Oil Boom To Gain Momentum In Second Half Of 2023

We had finally stopped the Corrupt Bastards Club and had gotten our state’s finances back on track with the ACES oil tax structure. We were building a large Constitutional Budget Reserve with ...

Letter: Oil tax rut

Alaska Dispatch News

Choke

Union of Soviet Socialist Republics

Group of 7 leaders are prepared to celebrate the results of a novel effort to stabilize global oil markets and punish Moscow. Send any friend a story As a subscriber, you have 10 gift articles to ...

An Untested Oil Price Cap Has Helped Choke Revenue to Russia

The New York Times

Get the most recent info and news about Apache on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Apache Corporation is an independent energy company. The Company explores for, develops, and produces natural gas, crude oil, and natural gas liquids. Apache operates worldwide.

Apache

Schemas are an essential part of any They’re metadata that define the shape of the data and the properties’ names and data types. Schemas are helpful in two ways. First, they provide a fixed blueprint for the data format, which can prevent badly formed data from being used within the context of the schema. Second, schemas let users understand how to parse the data and what to expect from its properties. data platform. is an open source, distributed publish-subscribe (pub-sub) messaging system that enables server-to-server data transportation . Messages are sent via topics, with the producer putting messages onto these topics to be read by the consumer. A user can define a schema for a topic stored in Pulsar’s schema registry. When messages are added to the topic, the Pulsar broker checks that the message conforms to the schema and ensures only valid messages are sent. The schema acts as a contract between the producer and consumers, enabling both parties to know the exact format of the data. Apache Pulsar using schema Over time, as applications evolve, it may be that the data produced by the application changes too. However, schema changes can affect the downstream consumers of the data that expect data in a specific format. Without a way to manage schemas between producers and consumers, it’s difficult to make changes to the data that is written in the messages or events without risking breaking downstream applications. To avoid this kind of issue, the schema of the messages must also evolve as new properties are added and allow consumers to understand data in both the old and new formats. This concept is known as schema evolution and Pulsar supports it. This article discusses why schemas evolve before diving into how Pulsar implements and supports schema evolution. Why Schemas Evolve Schemas provide context about raw data. They often describe a particular entity in a system, encompassing all the properties of that entity. For example, you may have an application that signs up users. It stores user details such as names, email addresses and ages. There would be a user schema that describes the underlying data and provides context, such as the name of the field and the type of data in it. The schema may look as follows: Now, say you wanted to expand the data captured to include address data and support direct mailing to the users. You would then need to expand the schema to include the new fields for capturing addresses, such as the first line of the address, city and ZIP code. After including these new fields, the schema will look like this: This is a simple form of schema evolution, as the original fields haven’t changed, and only new fields have been added. In most cases, this shouldn’t be a breaking change for downstream consumers, as consumers can continue as if the new fields were absent. The consumers would just need to be updated to consume and use the new properties. However, sometimes existing fields need to be amended to support new functionality. For example, say that users were uncomfortable giving an exact age, and instead, you change the application to capture age ranges like 18-24, 25-39,40-49 and 60+. The age column would need its data type amended from integer to string. This is a more complex schema evolution, as this could break downstream consumers who are processing the age property and expecting it to be a number or parsing the number using a strictly typed language like Java. They could also perform numeric calculations on the property, which would no longer work in its new format. To overcome this challenge, data platforms can support schema evolution to handle scenarios like this. Pulsar recognizes the importance of schema to data processing; in fact, it treats it like a first-class citizen by including built-in schema evolution support. Let’s look at how Pulsar does this. How Pulsar Supports Schema Evolution Pulsar defines schemas in a data structure called `SchemaInfo`. This is a Pulsar-specific data structure, which is itself a schema, that describes the schema of messages transported via Pulsar. Each `SchemaInfo` belongs to a topic (message channel) and describes the messages to be produced and consumed using that topic. Each `SchemaInfo` has a type that details the type of schema being used. This can be anything from an integer, a string, or a complex schema such as Avro or Protobuf. To support , Pulsar uses schema compatibility checks to compare an incoming schema to a topic’s known schema(s). Schema compatibility checks occur when a producer or consumer attempts to connect to a particular topic. The strategy is chosen while configuring the broker, with the value `schemaCompatibilityStrategy`. The goal is to check the `SchemaInfo` provided by the connecting client with existing `SchemaInfo` to see if they are compatible. schema evolution Pulsar supports eight different types of , which you can set based on the requirements of the change. These strategies also come with rules outlining what changes can be made and pointers around which clients to update first. (Check out the documentation linked above for a deeper explanation of each compatibility strategy). schema compatibility strategies Returning to the earlier examples, let’s implement the schema changes using Pulsar’s compatibility strategy. First, start with the initial user schema (without the address). This will be V1 of your schema. So, when you implement a Pulsar producer or consumer for the first time, the `SchemaInfo` for this version will be stored, and the producer and consumer will work as expected. Next, you want to add the new address fields to your user schema. The first step is to consult the schema compatibility strategies and determine which one is best for this change. Using the “changes allowed” column in the documentation, you’re looking for any strategy that allows the addition of new fields. This gives you BACKWARD, BACKWARD_TRANSITIVE, FORWARD, and FORWARD_TRANSITIVE. BACKWARD should be used when there’s no guarantee that consumers using an older version can understand the new schema. FORWARD is used when the consumers on the latest schema version might not be able to read data within older versions. If you want to update all of the consumers first to use the new schema, use a BACKWARD strategy. Otherwise, FORWARD is best. Looking at the bigger picture, Pulsar refers to the entire act of evolving a topic’s schema as . It’s the combination of a producer or consumer providing `SchemaInfo`, the chosen compatibility strategy for the topic, and the broker deciding what to do. schema verification Conclusion It’s rare for schemas to stay the same forever. As new features are introduced into applications, schemas often have to evolve to support these features. However, keeping the data's producers and consumers in sync can often be a challenge when schemas are modified. Pulsar’s built-in schema evolution concepts help deal with these changes. Using schema compatibility strategies, it can define the rules of how different compatible versions of a schema are. Pulsar uses this in conjunction with a schema verification process that then uses these rules to determine which schemas can be used by a consumer when connecting to a particular topic. Also published here.

How to Manage Schema Change with Apache Pulsar

Cómo administrar el cambio de esquema con Apache Pulsar

Comment gérer le changement de schéma avec Apache Pulsar

अपाचे पल्सर के साथ स्कीमा चेंज को कैसे प्रबंधित करें

Apache Pulsar を使用してスキーマ変更を管理する方法

Como gerenciar a mudança de esquema com o Apache Pulsar

Cách quản lý thay đổi lược đồ với Apache Pulsar

如何使用 Apache Pulsar 管理模式更改

Apache Cassandra is becoming the best database for handling JSON documents. If you’re a Cassandra developer who finds that statement provocative, read on. In a previous article, to mold Cassandra into a developer experience more idiomatic to the way developers think, thus improving developer productivity while preserving reasonable database performance and scale. It’s a great hypothesis and one that needs to be tested in the context of a particular developer idiom and developer community. we discussed using data APIs and data modeling , an object data mapping library usually used with the MongoDB driver, is an open source project with a significant community of JavaScript developers around it. At the , the open source API data gateway designed to work with Cassandra, we’ve partnered with Mongoose, and we’re working on an upcoming that will be released together with a that will work through that JSON API to connect to Cassandra. This creates an end-to-end stack for Mongoose developers that is fully open-source. It’s game-changing for Mongoose developers, and opens an important new chapter for Cassandra. Mongoose Stargate project JSON API version of Mongoose In this article, I’ll discuss how to provide a developer-friendly JSON idiom using Cassandra together with Stargate, and how we’re working to do just that for Mongoose developers. The Goldilocks of JS Communities In October 2022, we released a new version of Stargate. With the new , individual APIs are no longer embedded in the core Stargate coordinator code, but instead separated out into individual services. This improves Stargate’s operational efficiency; individual API services can now be deployed and scaled independently. This also makes new API services easier to develop. As long as they abide by the service boundary, these services can be developed in parallel with and independent of core Stargate development work. Version 2 We then looked for a truly idiomatic experience we could deliver to developers. With 18 million developers, JavaScript is the world’s most popular programming language, and JSON is the standard way that JavaScript developers structure data. However, 18 million people is not a community; it is many communities. We needed the “Goldilocks” of JavaScript communities — big enough to be significant, but small enough to be focused. We found the right community built around Mongoose, an object data mapper library used with applications that connect to MongoDB. Mongoose has several important characteristics: It’s JavaScript-centric It enjoys broad adoption, with roughly 2 million GitHub repositories listing Mongoose as a dependency Mongoose creator ’s active leadership provides a clear focus Valeri Karpov It’s an open source project that has lacked an open source database since MongoDB’s decision to move to a shared source model with its Server Side Public License Developers don’t really interact directly with a database so much as a data model. In Stargate’s original Document API, the API handles JSON by making it look like a traditional Cassandra table. This puts a burden on JSON-oriented developers to think in terms of Cassandra data structures, and puts a burden on Cassandra’s row-oriented indexing logic because a JSON document gets spread across multiple rows. Our new JSON API departs from this data model, and instead relies on a data model we call “super shredding.” You can learn more about super shredding by watching Aaron Morton’s talk at the recent event. In short, we take advantage of Cassandra’s wide-column nature to store one document per row, knowing that a Cassandra row can handle even very large documents. We also have a set of columns in that row that are explicitly for storing standard metadata characteristics of a JSON document. Now we have something more easily indexable, as well as a means of preserving and retrieving metadata. Cassandra Forward We will then front this data model with our new JSON API, using the same specification that Mongoose uses as our guiding requirement for which calls the API needs to support. When complete, this should enable any of the more than 2 million Mongoose-dependent applications to run against open-source Cassandra or DataStax’s hosted Cassandra service, , with just a configuration change. mQuery Astra DB With Mongoose and the new JSON API, we’ll provide a fully idiomatic experience to JSON-oriented JavaScript developers, giving them the underpinning an authentic JSON data model. scale and performance of Cassandra Mongoose creator Karpov also spoke at the recent Cassandra Forward event (you can watch the replay ), demonstrating a simple e-commerce application that uses the Stargate version of Mongoose, open source Stargate and the DataStax Enterprise (DSE) version of Cassandra. You can download the working code for this application and the supporting platform pieces from . While we have enough code to run this application, we are not yet code complete. For example, we run against DSE right now because we need (SAI), which works with DSE and is planned for release in Cassandra 5.0 later this year. here GitHub storage-attached indexing Contributing Back to Cassandra Cassandra isn’t a static piece of software; it’s a vibrant and evolving open-source project. So we are also continuing a longstanding Cassandra tradition of using that emerge client-side to foster changes on the database side.o  Similarly, Stargate’s Mongoose work has prompted a set of proposals for Cassandra around global sort and advanced query filtering that will not only make Stargate’s JSON API and Mongoose client better, but will add powerful new features to Cassandra Query Language. This is a great reminder that data engineers and application developers are not two different communities, but complementary cohorts of the extended Cassandra community. features like SAI And JSON is just the first step. Essentially, what we will have done is to take the building blocks of Cassandra, Stargate and a reasonably efficient Cassandra data model and build a document database that you interact with through a JSON API. In other words, we’ve used super shredding to create a purpose-built database that better serves the community of Mongoose developers. With the modular architecture of , and the proof point of Mongoose for the idiomatic approach, we are ready to take on new developer communities that organize around a particular software development idiom. The process by which we’ve harnessed Cassandra for Mongoose is repeatable — and it’s one that we will repeat. In so doing, we dramatically expand the number of developers and use cases that Cassandra can address, which is the sort of goal worthy of an open-source project. Stargate v2 . Learn more about DataStax By Mark Stone. Mark is product manager at DataStax. He's a technology veteran with many years of experience in product management, program management and people management. Always working as part of the connective tissue between business stakeholders and technical stakeholders, Mark loves championing the developer experience in technology platforms and helping organizations meet developers where they are.

How Mongoose Will Bring JSON-Oriented Developers to Apache Cassandra

Cómo Mongoose traerá desarrolladores orientados a JSON a Apache Cassandra

Comment Mongoose amènera les développeurs orientés JSON vers Apache Cassandra

Mongoose JSON-उन्मुख डेवलपर्स को Apache कैसेंड्रा में कैसे लाएगा

Mongoose が JSON 指向の開発者を Apache Cassandra にもたらす方法

Como a Mongoose trará desenvolvedores orientados a JSON para o Apache Cassandra

Mongoose sẽ đưa các nhà phát triển định hướng JSON đến với Apache Cassandra như thế nào

Mongoose 如何将面向 JSON 的开发人员带到 Apache Cassandra

This article explains an alternate option for Apache Airflow to manage or run data quality checks. is an open-source platform that has become increasingly popular for managing data pipelines. Airflow And why not, it provides a convenient way to automate and organize the workflow of tasks related to data processing, making it easier for organizations to manage their data pipelines efficiently. However, while Airflow has proven to be a powerful tool for managing data pipelines, it has significant limitations regarding managing data quality. Needless to say, compromised data quality is of no good to the organization. In this blog, we’ll discuss some challenges while using Airflow and why you might need to reconsider this choice for data quality-related tasks. By the end of the blog, we hope you’ll determine if Airflow is the right fit for your data needs. Data quality is an essential aspect of any data pipeline, and it's critical to ensure that data is accurate, consistent, and free from errors. The data quality must be monitored and maintained throughout the entire data pipeline, from the source to the final destination. Interestingly, Airflow does not have built-in features for managing data quality, so organizations must use additional tools and techniques to maintain the quality of their data. Let us discuss some of Airflow's limitations when managing data quality and why it's not the best choice for organizations that need to ensure the accuracy and consistency of their data. We’ll also discuss alternative tools and techniques organizations can use to manage data quality in their pipelines. By understanding the limitations of Airflow and the importance of data quality management, organizations can make informed decisions about how they manage their data pipelines and ensure that they use the right tools for the job. So, let us see why Airflow might not be the best choice for your data quality needs. Reasons Not to Use Airflow for Data Quality: 1. Lack of Built-in Data Quality Control Features: Airflow was designed to automate and manage data pipelines, but it does not have built-in features for managing data quality. Without these features, it becomes difficult to detect errors and inconsistencies in the data, leading to incorrect insights and decision-making based on inaccurate data. This would mean that organizations must use additional tools to validate their data and ensure that it's accurate and consistent. 2. Inefficient Data Validation Processes: Data validation is a crucial step in data quality management. This leads to a higher risk of errors and inconsistencies in the data, which can compromise the accuracy and consistency of the data. The inefficient data validation processes in Airflow can also lead to a longer time to complete data processing tasks, impacting the organization's ability to make timely and informed decisions. 3. High Risk of Data Errors and Inconsistencies: The lack of built-in data quality control features and the inefficient data validation processes in Airflow increases the risk of data errors and inconsistencies. These errors and inconsistencies can compromise the accuracy and consistency of the data, leading to incorrect insights and decision-making based on inaccurate data. This can have serious consequences for the organization, including lost revenue, damaged reputation, and decreased customer trust. 4. Data lineage is an important aspect of data quality management as it provides a clear understanding of the origin, transformation, and flow of data throughout a pipeline. With Data lineage, data engineers derive the what, where, and how of the data. What data is being processed, where it comes from, and how it's transformed? In Airflow, the lack of built-in lineage information can make it difficult for data engineers to fully understand the impact of any changes to data quality checks or the data pipeline as a whole. This can make it difficult to trace the root cause of data quality issues. Furthermore, without lineage information, data engineers may struggle to identify dependencies between different data quality checks and the impact of changing those checks. This can lead to errors in the data pipeline and negatively impact data quality. Now that we understand its incapability in data lineage and the overall data quality, let us see the major challenges of using Airflow in a distributed system. Challenges in Managing Data Quality in a Distributed System: Data quality management in distributed systems presents unique challenges that require specialized features. Some of the challenges include the following: Lack of coordination across distributed teams: Airflow does not have the built-in capabilities to manage the coordination of data quality checks across multiple teams. Imagine a software company with a distributed data team responsible for different data sources, such as user data, product data, and usage data. Each department has its own data quality checks and procedures in place, but there is no centralized tool or process to coordinate data quality management across the teams. This results in data quality issues slipping through the cracks and inconsistent data quality across the different data sources. Real-time data quality management: Airflow may not have the ability to keep up with the speed and scale of data processing in these systems, leading to gaps in data quality management. A SaaS company that processes large amounts of user data in real time would need robust speed and scale for data processing. Airflow is not capable of keeping up with this. As a result, data quality issues are not identified and corrected in a timely manner, leading to gaps in data quality management. Inconsistency managing multiple data sources: Airflow may not have the necessary tools to manage data quality consistency across multiple data sources in a distributed system. A software company has multiple data sources, including user data from its SaaS platform, product data from its APIs, and usage data from its analytics tools. Each data source has different data quality standards, making it difficult to maintain consistent data quality across all sources. Data quality Issues while handling distributed systems: Airflow may not have the built-in tools to efficiently address data quality issues in a timely manner. When data quality issues are identified, fixing them can be difficult and time-consuming, especially in a system with multiple teams and data sources. Airflow does not have the built-in tools to efficiently address data quality issues in a timely manner, making it a challenge to maintain high-quality data in the system. Airflow is not designed for data quality. Let's see which alternatives can assist you if you are looking for a solution to all these challenges. Better Alternatives to Ensure Data Quality: There are several alternative tools and techniques that organizations can use to manage data quality in their data pipelines. These tools are specifically designed for data quality management and provide data validation, profiling, and cleansing features. It's all about choosing the one that suits best your needs. Here we suggest the most popular alternatives: Talend provides a user-friendly interface with drag-and-drop functionality, making it easy for data engineers to create and execute data quality checks. It also offers a range of data quality tools to ensure accurate and consistent data and can be integrated with multiple data sources, unlike Airflow. Microsoft SQL Server Integration Services (SSIS), with scalability and performance to handle large volumes of data, offers built-in data quality tools for cleaning, standardizing, and validating data. Its capability to seamlessly integrate with top technologies such as Azure, and Power BI makes it the go-to choice for organizations seeking to maximize their data capabilities while minimizing integration challenges. Our favorite: dbt-test! It's a testing framework specifically designed for data engineers and data scientists. It's simple to use, with a minimal learning curve, and provides a range of tests to validate data quality and accuracy. Unlike Airflow, dbt-test is designed specifically for data quality management, making it a more efficient and effective solution. With its ability to handle complex data quality scenarios, dbt-test is the preferred choice for many organizations looking to manage data quality in their data pipelines. Our favorite is the combination of Airflow and dbt-test. Why you may ask. Airflow and dbt together can enable each member of the data team to focus on their specific areas of expertise and have a clear understanding of who should be involved in any troubleshooting when issues arise. How does that happen? Airflow and dbt each serve a specific purpose in the data engineering workflow, with Airflow being responsible for orchestrating jobs related to data extraction, loading, and machine learning processes, while dbt focuses on data transformation using SQL. When used together, these two tools can complement each other to provide the best results for data teams. ‍ So, always evaluate the alternatives and make a wise decision in choosing the right tool since it's going to make or break the data quality. Adapting the Right Tool Resolves Data Quality Issues: While Airflow has some features that make it useful for data engineering, it's not the best choice for data quality management. The lack of built-in tools and the difficulty in coordinating and managing data quality across distributed teams make it an inefficient solution for data quality management. Alternatives provide more comprehensive data quality management capabilities, making them a better choice for data quality management in distributed systems. Airflow with dbt-test, in particular, stands out for its simplicity and specificity to data quality management, making it an efficient and effective solution for organizations since this combination enables data teams to work collaboratively and deliver reliable data to end-users, ensuring high-quality data in their pipelines. What steps are you taking to ensure bullet-proof data quality in your distributed systems? Also published here

Apache Airflow: Is It a Good Tool for Data Quality Checks?

An extraordinary computer science breakthrough called Accord is bringing globally available, general-purpose ACID transactions to the next Cassandra release Let’s get the best part out first. are coming to Apache Cassandra. ACID transactions Globally available, general-purpose transactions that work the way Cassandra works. This isn’t some trick with fine print or the application of old techniques. It’s due to an extraordinary computer science breakthrough called from a team at Apple and the University of Michigan. It will help Cassandra change how we think of data by opening new use cases. Accord Here’s what this means for those who aren’t steeped in the ins and outs of the Cassandra project and its features. Nothing is more important than how fast you can put an application into production. But developers that want to put Cassandra’s scale, resilience, and legendary multi-data center support to work on something like financial transactions had to code a bunch of complicated workarounds into their apps. The trade-offs versus using, say, Oracle were significant. With Accord? No trade-offs. Cassandra will now that has made it amazing while shifting transaction support to the database, significantly reducing code complexity. support everything The Eye of the Observer A database system has essential functions, such as and being available for query. Managing changes in data isn’t always a database function. In the case of many NoSQL systems, the burden of change management is deferred to the user. The observer of the data change is who assigns the importance of exclusivity. storing data reliably Suppose the point is to accumulate data as given. In that case, the observer must know the data was received and stored durably — for example, stock ticker data where every data point is unique and cumulative. There is no need for exclusivity. In more sensitive operations, the observer of a data change needs to feel like they are the only process using the database. It’s a concept in computer science called “isolation”; it’s the “I” in ACID  (atomicity, consistency, isolation, durability). A classic example is a bank transfer where money is subtracted from one bank account and then added to another — in precisely that order. The observing process needs exclusivity to avoid other processes making changes that can lead to inconsistencies or surprises. Surprises include inadvertently allowing a transfer from an account that has gone below $0. Isolation guarantees only one process can make a change at a time, and if two are competing for the same data, one of them will have to wait for the other to complete. Embrace Your Laziness Developers need to move quickly with a system they can trust. ACID transactions have been the gold standard of trust within database systems for almost 50 years. Developers work through trade-offs based on requirements, sometimes leading them to work with systems that don’t support ACID transactions. With NoSQL systems, trade-off biases historically tended to tip toward scale and uptime while sacrificing transactions. Bringing ACID transactions to Cassandra has been about reducing trade-offs. Cassandra already has a solid reputation for linear scaling while maintaining uptime in the worst scenarios. Cassandra has been the choice when you need a database that will hold up to what the internet can deliver. Unsurprisingly, the need for transactions has been a source of trade-off conflict for developers. Can We Get a Consensus? In distributed systems, each member node in the larger cluster can act independently or need to coordinate with other nodes. In a transaction where, “Hey, we all need to agree on something,” computer scientists call this consensus, and developing those protocols is a continuous area of improvement. has been a long-established consensus protocol adopted by Cassandra in 2013 for “lightweight transactions.” Paxos Lightweight because it ensures that a single partition data change is isolated in a transaction, but more than one table or partition is not an option. In addition, Paxos requires multiple round trips to gain a consensus, which creates a lot of extra latency and fine print about when to use lightweight transactions in your application. The protocol was developed as the next generation to replace Paxos, and several systems such as Etcd, CockroachDB, and DynamoDB adopted it. It reduced round trips by creating an elected leader. Raft The downside for Cassandra in this approach is that leaders won’t span data centers, so multiple leaders are required (see ). Spanner Having an elected leader also violates the “shared-nothing” principles of Cassandra and would layer new requirements on handling failure. If a node goes down, a new leader has to be elected. Other databases — FaunaDB and FoundationDB, for example — have gone down the route of trying to solve the multileader problem by reducing down to a single, global leader, as described in the . Calvin paper Because these were built for other databases with different requirements, the approaches used in those cases failed to meet the criteria Cassandra expects with failure modes. Cassandra assumes failures as a part of running an extensive distributed system. One or more nodes going offline should not cause rapid performance degradation or availability issues. We needed a different approach. Have We Reached an Accord? We can get very opinionated on what is acceptable for the Cassandra project. Our criteria are about holding true to the core beliefs on how distributed systems should run. Performance and scaling should always be preserved while operating multiple nodes across one or more data centers. We can be pretty demanding, but this is what makes Cassandra the choice for so many organizations. The previous iterations of consensus protocols solved different parts of the problem, but each presented a trade-off that would violate some of Cassandra’s values. It’s been said the next big breakthrough is two papers away from the last. In this case, the paper was , and it took a big swipe at eliminating tradeoffs. Accord Accord addresses two problems that aren’t solved in previous consensus protocols: How can we have a globally available consensus and achieve it in one round trip? The first novel mechanism is the reorder buffer. Assuming commodity hardware is in use, differences in clocks between nodes are inevitable. The reorder buffer measures the difference between nodes in addition to the latency between them. Each replica can use this information to correctly order data from each node and account for the differences, guaranteeing one round-trip consensus with a timestamp protocol. The other mechanism is fast-path electorates. Failure modes can create latency when electing a new leader before resuming. Fast-path electorates use pre-existing features in Cassandra with some novel implementations to maintain a leaderless fast path to quorum under the same level of failure tolerated by Cassandra. More details can be read in the . proposal How Does It Work? The biggest impact will be in developer productivity, so let’s see what that looks like in practice. Consider the following bank account transfer example we mentioned earlier: First is the new syntax you’ll see in the Cassandra Query Language (CQL). Transactions are contained in a and declaration. Everything inside the transaction markers will happen atomically in isolation from other processes. We will transfer $20 from Alice’s account to Bob in this example. It doesn’t get any more classic than that! BEGIN TRANSACTION COMMIT TRANSACTION In section A, we can select data from an existing record and assign the result to a tuple (multiple items stored in a single variable). Depending on how many columns are in the clause, you can store one or more values in the tuple. These values will be used in section B to test conditions before making data changes. SELECT In this case, we will test to see if Alice has $20 in her account before transferring to Bob. If so, then an decrements Alice’s account balance by $20 and then increments Bob’s by $20. If Alice had less than $20, then the changes wouldn’t happen. UPDATE Behind the scenes is a serialized set of database commands that execute exclusively, as seen from the observing process. Across one or more data centers, the transaction only required one round trip to gain consensus, and if any nodes were offline, the action would still occur if at least a quorum of replicas were available. This is exactly how Cassandra likes to work, but we just upped our game with a globally available transaction. What’s Next Accord and all of the work that goes with it is still in progress and slated to be included in the next Cassandra release. Since this is all in open source, those of you that can’t wait can clone a copy of the branch from the Cassandra repository and build your own copy. cep-15-accord For the rest of you, as we get closer to release time, we’ll have builds available for you to use and test. It will be a game-changer for Cassandra, and I’m sure you’ll want to see it for yourself. I’m most interested in hearing from the community about what use cases you’ll find with a globally available transaction running at the speed and resilience you expect from Cassandra. Is it finally time to let go of those last relational database workloads? We’re also anxious to hear your feedback on all our channels, including the Apache Software Foundation . Features in an open source project are constantly evolving to meet users’ needs. That’s why you’ve got a critical role to play in shaping for the future. Slack or project mailing list Apache Cassandra And stay tuned for more use cases and information as we evolve this exciting new feature. You can expect that there will be several talks about this at the digital summit coming up on March 14. You won’t want to miss those. Cassandra Forward By Patrick McFadin, DataStax Patrick McFadin is the co-author of the O’Reilly book 'Managing Cloud Native Data on Kubernetes.' He currently works at DataStax in developer relations and as a contributor to the Apache Cassandra project. Patrick has worked as chief evangelist for Apache Cassandra (he’s also a newly minted Cassandra committer!) and as a consultant for DataStax, where he had a great time building some of the largest deployments in production.

ACID Transactions Are Coming To Apache Cassandra: Here's Why We're Excited

Las transacciones ACID están llegando a Apache Cassandra: He aquí por qué estamos emocionados

Les transactions ACID arrivent sur Apache Cassandra : voici pourquoi nous sommes ravis

एसिड लेनदेन अपाचे कैसेंड्रा में आ रहे हैं: यही कारण है कि हम उत्साहित हैं

ACID トランザクションが Apache Cassandra に登場: これが私たちが興奮する理由です

As transações ACID estão chegando ao Apache Cassandra: eis por que estamos entusiasmados

Các giao dịch ACID đang đến với Apache Cassandra: Đây là lý do tại sao chúng tôi rất phấn khích

ACID 事务即将进入 Apache Cassandra：这就是我们兴奋的原因

API Gateways are critical components in one's infrastructure. If an attacker could change the configuration of routes, they could direct traffic to their infrastructure. Consequences could range from data theft to financial losses. Worse, data theft could only be noticed after a long time by mirroring the load. Hence, protecting your API Gateway is of utmost importance. In this short blog post, I'll list a couple of ways to secure your Apache APISIX admin access. Change Admin Tokens You can manage Apache APISIX configuration via its HTTP APIs. A token protects every API call. Operations require an HTTP Header: X-API-KEY Use a token with the role to call read operations viewer Use a token with the role to call read write operations admin and For example, to create a new route, I need to pass an -role token, which allows calling write operations: admin curl http://localhost:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
  "methods": ["GET"],
  "uri": ["/hello"],
  "upstream_id": 1
}' The first and foremost step to secure your access is to change the default token values: deployment:
  admin:
    # Default token when use API to call for Admin API.
    # *NOTE*: Highly recommended to modify this value to protect APISIX's Admin API.
    # Disabling this configuration item means that the Admin API does not
    # require any authentication.
    admin_key:
      - name: admin
        key: edd1c9f034335f136f87ad84b625c8f1                                    #1
        role: admin                 # admin: manage all configuration data
                                    # viewer: only can view configuration data
      - name: viewer
        key: 4054f7cf07e344346cd3f287985e76a2                                    #1
        role: viewer Change it! You may want to secure tokens even further; it depends on your platform. For example, you may want to store tokens as and inject them at container startup. Secret Restrict Binding IP(s) A server can have multiple IPs from different network adapters. For example, an API Gateway would have at least two network adapters: One public-facing adapter to be reachable from the Internet One internal for inside access By default, Apache APISIX will bind itself to all network adapters found on the server at startup. The above scenario means it will be reachable . We should restrict access from the inside only. from the Internet We can set which network interface Apache APISIX can bind to in the configuration: deployment:
  admin:
    admin_listen:
      ip: 0.0.0.0     # Specific IP, if not set, the default value is `0.0.0.0` #1 Change it! Restrict Allowed IPs Even if you restrict access to only IPs from inside your enterprise network, you want only some machines to access the API Gateway configuration. If it was the case, an attacker gaining access to the machine of an accountant could use it to try to attack the API Gateway. You can restrict IP access with network policies - and you should. However, you can also implement this restriction on the API Gateway: it can allow finer-grained control and more agile changes - network policies are hard to change in general. Here's the relevant snippet for Apache APISIX: deployment:
  admin:
    allow_admin:
      - 127.0.0.0/24 # If we don't set any IP list, then any IP access is allowed by default
      #- "::/64"                                                                #1 Change it according to your network topology Mutual TLS If to talk about authentication, one way is via a digital certificate. The most widespread low-level authentication mechanism today is Transport Layer Security. TLS allows servers to prove their identity. Additionally, it keeps data exchanged private and prevents them from being tampered with. Mutual TLS works on both sides so that the server proves its identity to the client, the client proves its identity to the server. and Here's the relevant configuration snippet to set admin mTLS in Apache APISIX: deployment:
  admin:
    https_admin: true   # enable HTTPS when use a separate port for Admin API
                        # Admin API will use conf/apisix_admin_api.crt and conf/apisix_admin_api.key as certificate
    admin_api_mtls:
      admin_ssl_ca_cert:  "/data/certs/mtls_ca.crt"       # Path of your self-signed ca cert
      admin_ssl_cert:     "/data/certs/mtls_server.crt"   # Path of your self-signed server side cert
      admin_ssl_cert_key: "/data/certs/mtls_server.key"   # Path of your self-signed server side key Standalone Mode Last but not least, one can completely move the configuration from to a static (YAML) file. In this case, no Admin API is available to update the configuration. This deployment model is known as . etcd standalone mode In standalone mode, the only way to change the configuration is to update the static file: Apache APISIX will check for changes at regular intervals and update itself accordingly. It's a great way to apply GitOps principles with Apache APISIX. In the meanwhile, you can configure the standalone mode as the following: deployment:
    role: data_plane
    role_data_plane:
       config_provider: yaml Note that standalone mode makes all other securing options moot, as there's no Admin API to secure anymore. Conclusion You need to secure API Gateways from unwanted access as they are valuable targets for bad actors. In this post, I've shown several non-exclusive options you should consider to secure Apache APISIX. Finally, I've described how you could make it without the Admin API via the dedicated Standalone deployment mode. Expect a future blog post on how to use it with GitOps. To go further: mTLS: Protect Admin API Standalone mode Originally published at on February 5th, 2023 A Java Geek

How to Secure Admin Access to Apache APISIX

Apache APISIX, the Apache-led API Gateway, comes out of the box with many plugins to implement your use case. Sometimes, however, the plugin you're looking for is not available. While creating your own is always possible, it's sometimes necessary. Today, I'll show you how to route users according to their location without writing a single line of Lua code. Why geo-routing? Geo-routing is to forward HTTP requests based on a user's physical location, inferred from their IP. There are many reasons to do that, and here is a couple of them. Note that I'll use the country as the location-dependent factor, but any smaller or bigger scale works. It's the scale I'm most familiar with - and probably the most useful. First, most applications are not meant to be geo-dependent. The app your team has just developed probably only makes sense in a single country, if not a single region. In this case, geo-routing will never be a problem. However, some apps do grow along with the business. When it happens, the need for appears. It's the app's responsibility to handle such geo-dependent factors. i18n should be handled natively by the tech stack, e.g., in . l10n is more ad hoc but shouldn't be a problem either. internationalization and localization Java Issues arise when business rules diverge from country to country, chiefly because of laws. Other reasons include a partnership. Imagine an e-commerce shop that has branches in many countries. You may choose the delivery partner, but depending on the country, available partners are different. While keeping a single codebase is always a wise choice, even the best design can only slow down the chaos from many business rules. At one point, splitting the God app into multiple country-dependent apps is inevitable. Sometimes, you don't even have a choice. A country decides you have to store your database on their territory, so you cannot share it anymore and have to split both storage and app. I witnessed it first-hand with Russia in 2015: we had to deploy a custom version of our e-commerce application just for Russia. Finally, you may also want to deploy a new app version for a single country only. In this case, you should monitor not (only) technical metrics but business ones over time. Then you'll decide whether to expand the new version to other countries based on them or work more on the latest version before deploying further. Setting up Apache APISIX for geo-routing Though I'm a developer by trade (and passion!), I'm pragmatic. I'm convinced that every line of code I don't write is a line I don't need to maintain. Apache APISIX doesn't offer geo-routing, but it's built on top of Nginx. The latter provides a geo-routing feature, albeit not by default. The following instructions are based on Docker to allow everybody to follow them regardless of their platform. We need several steps to set up geo-routing on Apache APISIX: Create a custom Docker image Add the required library module Add its dependencies Configure Apache APISIX Nginx geo-routing requires the module. But if we try to install it via a package manager, it also installs , which conflicts with the instance embedded in Apache APISIX. As we only need the library, we can get it from the relevant Docker image: ngx_http_geoip_module nginx nginx FROM nginx:1.21.4 as geoiplib

FROM apache/apisix:2.15.0-debian

COPY --from=geoiplib /usr/lib/nginx/modules \
    /ngx_http_geoip_module.so /usr/local/apisix/modules/ngx_http_geoip_module.so      #1 Copy the library from the image to the one nginx apache/apisix The regular package install installs all the dependencies, even the ones we don't want. Because we only copy the library, we need to install the dependencies manually. It's straightforward: RUN apt-get update && apt-get install -y libgeoip1 Nginx offers two ways to activate a module: via the command line or dynamically in the configuration file. The former is impossible since we're not in  control, so the latter is our only option. To update the Nginx config  file with the module at startup time, Apache APISIX offers a hook in its  config file: nginx.conf nginx_config:
  main_configuration_snippet: |
    load_module     "modules/ngx_http_geoip_module.so"; The above will generate the following: # Configuration File - Nginx Server Configs
# This is a read-only file, do not try to modify it.
master_process on;

worker_processes auto;
worker_cpu_affinity auto;

# main configuration snippet starts
load_module     "modules/ngx_http_geoip_module.so";

... The relies on the . We installed it implicitly in the previous step; we have to configure the module to point to it: GeoIP module Maxmind GeoIP database nginx_config:
  http_configuration_snippet: |
    geoip_country   /usr/share/GeoIP/GeoIP.dat; From this point on, every request going through Apache APISIX is geo-located. It translates as Nginx adding additional variables. As per the documentation: The following variables are available when using this database: two-letter country code, for example, , . $geoip_country_code "RU" "US" three-letter country code, for example, , . $geoip_country_code3 "RUS" "USA" country name, for example, , . $geoip_country_name "Russian Federation" "United States" - - Module ngx_http_geoip_module Testing geo-routing You may believe that the above works - and it does, but I'd like to prove it. I've created whose architecture is simple: a dedicated project Apache APISIX configured as above Two upstreams, one in English and one in French upstreams:
  - id: 1
    type: roundrobin
    nodes: "english:8082": 1
  - id: 2
    type: roundrobin
    nodes: "french:8081": 1
routes:
  - uri: /
    upstream_id: 1
  - uri: /
    upstream_id: 2
#END With this snippet, every user accesses the English upstream. I intend to direct users located in France to the French upstream and the rest to the English one. For this, we need to configure the second route: routes:
  - uri: /
    upstream_id: 2
    vars: [["geoip_country_code", "==", "FR"]]   #1
    priority: 5                                  #2 The magic happens here; see below. By default, route matching rules are evaluated in arbitrary order. We need this rule to be evaluated first. So we increase the priority - the default is 10. Most Apache APISIX users are used to matching on routes, methods, and domains, but there's . One can match on Nginx variables, as shown above. In our case, the route matches if the variable is equal to . more to it geoip_country_code "FR" Note that values readability over power. Use the attribute if you need more complex logic. vars filter_func(vars) We can still not test our feature at this point, as we would need to change our IP address. Fortunately, it's possible to cheat (a bit), and the cheat is helpful in other scenarios. Imagine that Apache APISIX is not directly exposed to the Internet but sits behind a reverse proxy. There might be multiple reasons for this: "history", a single RP pointing to multiple gateways under the responsibility of different teams, etc. In this case, the client IP would be the RP's proxy. To propagate the original client IP, the agreed-upon method is to add an request HTTP header: X-Forwarded-For The X-Forwarded-For (XFF) request header is a de-facto standard header for identifying the originating IP address of a client connecting to a web server through a proxy server. When a client connects directly to a server, the client's IP address is sent to the server (and is often written to server access logs). But if a client connection passes through any forward or reverse proxies, the server only sees the final proxy's IP address, which is often of little use. That's especially true if the final proxy is a load balancer which is part of the same installation as the server. So, to provide a more-useful client IP address to the server, the X-Forwarded-For request header is used. - X-Forwarded-For The Nginx module offers this configuration but restricts it to an IP range. For testing, we configure it to any IP; in production, we should set it to the RP IP. nginx_config:
  http: geoip_proxy     0.0.0.0/0; We can finally test the setup: curl localhost:9080 {
  "lang": "en",
  "message": "Welcome to Apache APISIX"
} curl -H "X-Forwarded-For: 212.27.48.10" localhost:9080    #1 is a French IP address 212.27.48.10 {
  "lang": "fr",
  "message": "Bienvenue à Apache APISIX"
} Bonus: Logs and Monitoring It's straightforward to use the new variable in the Apisix logs. I'd advise it for two reasons: At the beginning to make sure everything is ok In the long run, to monitor traffic, e.g., send it to Elasticsearch and display it on a Kibana dashboard Just configure it accordingly: nginx_config:
  http:
    access_log_format: "$remote_addr - $remote_user [$time_local][$geoip_country_code] $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" $upstream_addr $upstream_status $upstream_response_time" #1 Keep the default log variables and add the country code Conclusion Geo-routing is a requirement for successful apps and businesses. Apache APISIX doesn't provide it out-of-the-box. In this post, I showed how it could still be straightforward to set it up using the power of Nginx. You can find the source code for this post on . GitHub To go further: Module ngx_http_geoip_module Converting Static Modules to Dynamic Modules Customize Nginx configuration GeoIP Update Originally published at on November 6th, 2022 A Java Geek

How to Configure Apache APISIX For Geo-routing

At , we use to host websites like our , our , and our . While this doesn’t mean we have to use Apache to run the sites locally, using a similar stack for development and production is generally a good idea. This means setting up a development environment with Apache. Tower Apache learning platform blog main product site On macOS, which is what we use, for the most part, there are quite a few options. We could set up Apache in a virtual machine, perhaps controlled using . We could use to run Apache in a container. There are also solutions with graphical user interfaces like . However, a convenient and simple solution is to just set up Apache running natively in macOS — no wrappers, no virtual machines, no containerization. In this post, we’ll go through how to set up Apache and PHP, using versions installed using the for macOS. Vagrant Docker MAMP Homebrew package manager macOS comes with built-in versions of Apache and PHP, and we could easily use those. However, there are a few drawbacks to this approach. We don’t have control over the exact versions used, and the version available might not be up to date. I’ve had problems where OS updates have overwritten my configuration for the built-in Apache server. Finally, running to check the version of the built-in PHP gives a warning message stating that PHP will be removed from future versions of macOS — in fact, in the upcoming macOS Monterey, PHP . Instead of using the built-in versions, we’ll install Apache and PHP using Homebrew. php -v seems to be gone We'll look at how to set up a local development environment using these and, as usual, we’ll try to cover the “why” as well as the “what”; in addition to just presenting the configuration, we'll go over the purpose of each directive and command. Here are the steps we'll take: Installing Apache and PHP Configuring Development URLs Apache Configuration PHP Configuration Virtual Host Setup Up and Running 1. Installation Instructions for how to install Homebrew itself can be found on . Assuming Homebrew is installed, all you need to do in order to install Apache and PHP is to run the following command: ( refers to the Apache webserver). the official Homebrew website brew install httpd php httpd A word regarding paths: on a Mac with Apple Silicon, Homebrew will use as its prefix. The prefix is sort of a base directory; a directory under which Homebrew will put various files belonging to the packages it installs. Binary files will go in , configuration files in , and so on. On an Intel-based Mac, this base directory will likely be instead. /opt/homebrew /opt/homebrew/bin /opt/homebrew/etc/ /usr/local In this article, I’ll refer to paths using a Homebrew prefix of . If your prefix is different, please change the paths accordingly. To find out which base directory Homebrew is using on your machine, run . /opt/homebrew brew --prefix 2. Development URLs Before we get started on Apache and PHP configuration, let’s touch on the topic of development URLs briefly. I think a nice setup for local development is to use a specific TLD like , accessing my various projects through URLs like , , and so on. In this guide we’ll simply use the file to point our “fake” domains at our local web server. If there’s interest, a future post might cover how to set up a service like to automatically grab all requests to hosts ending with and send them to our local development server. .test my-first-project.test my-second-project.test /etc/hosts dnsmasq .test is a reserved top-level domain and so, using this, you shouldn’t run into problems such as the people using for development environments did when TLD. .test .dev Google acquired the .dev The file provides a convenient way to associate hostnames with IP addresses. Normally, when we visit a URL like , the actual IP address of the server has to be determined through something called the Domain Name System, or DNS for short. The file gives us an easy way to override this. /etc/hosts https://www.git-tower.com/ /etc/hosts For the purposes of this article, let’s say we’re working with a project called , which we want served at the URL . We’ll add this line at the bottom of : my-project my-project.test /etc/hosts 127.0.0.1 my-project.test After saving the file, visiting in a browser will result in the request being sent to the IP address for our own machine. my-project.test 3. Apache Configuration Next, let’s get to work on the actual Apache configuration. In my case, the main Apache configuration file is located at (again, on an Intel-based Mac, this is likely to be ). In this file, there are a few changes to make: /opt/homebrew/etc/httpd/httpd.conf /usr/local/etc/httpd/httpd.conf Listen 8080 This line tells Apache to listen for traffic on the port 8080. Accessing ports with numbers lower than 1024 require root privileges and so, listening on port 8080 lets users run Apache without being root. However, as HTTP traffic goes to port 80 by default, we want to listen on that port instead: Listen 80 Chances are, you want to run multiple websites on your computer, with several hostnames in . However, all of these will hit the same Apache server. To serve multiple sites from one Apache server, Apache can look at the hostname of the incoming request and pass the request to one of multiple . Virtual host support has to be enabled by removing the in front of the line below, turning it from a comment into an Apache directive that loads the virtual hosts configuration from the file in question: /etc/hosts virtual hosts # #Include /opt/homebrew/etc/httpd/extra/httpd-vhosts.conf We’ll uncomment another line in order to load the module. This module is used to rewrite incoming URLs. For example, many web frameworks use it to enable “pretty URLs”, letting site visitors use URLs like while translating them into URLs like for the back-end. This module will likely be useful, so let’s enable it by removing the below: mod_rewrite /posts/2021/some-post-title/ /index.php?p=697 # #LoadModule rewrite_module lib/httpd/modules/mod_rewrite.so Apache comes with configuration for a default site, with a document root of . We won’t be using this site, as we’ll use virtual hosts instead. As it stands, everything would work, but Apache would emit a warning every time it started, stating that it can’t determine the hostname to use for this default site. We’ll set the hostname on a per-virtual host basis later. However, to silence the warning message, we’ll uncomment the default directive in (or just enter any hostname you want): /opt/homebrew/var/www ServerName httpd.conf ServerName www.example.com:8080 4. PHP Configuration We want PHP to be available on our server. For this, we’ll add another directive after the other ones, loading a module provided by PHP as installed through Homebrew earlier (on an Intel-based Mac, change to ): LoadModule /opt/homebrew /usr/local LoadModule php_module /opt/homebrew/opt/php/lib/httpd/modules/libphp.so This loads the PHP module. There is some additional configuration required. Let’s add further PHP configuration in a separate file in the directory, just like with the virtual hosts configuration. We need to include it explicitly from the main configuration file, so find the section with Include directives and add this after the last one: extra # PHP settings
Include /opt/homebrew/etc/httpd/extra/httpd-php.conf The rest of our PHP configuration goes in the file : /opt/homebrew/etc/httpd/extra/httpd-php.conf <IfModule php_module> 
  <FilesMatch .php$>
    SetHandler application/x-httpd-php
  </FilesMatch>

  <IfModule dir_module>
    DirectoryIndex index.html index.php
  </IfModule>
</IfModule> We start by checking if the PHP module is available, which might seem redundant as we just added it in the file. However, these files may get edited independently of each other, so let’s follow the convention of the other files in the extra directory and check that the modules we use are available. We’ll look for files ending in , and set their handler to — a handler provided by the PHP module. A handler in Apache represents an action to take for a file. While most files are just served using a built-in handler, the PHP files have to be interpreted by PHP before being served. httpd.conf .php application/x-httpd-php Some guides use the directives here, which take parameters for the file extensions they apply to. These can as they check their configured extensions against every extension of a file. For example, a web application may allow users to upload files. However, if a user uploads a file with a name ending in , this file might be executed as PHP if the above directives are used. Therefore, we use the directive along with . Not that we're likely to encounter these issues when we're running Apache locally for ourselves, but we might as well set things up properly. AddType or AddHandler introduce security issues .jpeg .php.jpeg FilesMatch SetHandler The directive makes sure that if a URL for a directory is requested, and the directory contains an file (or an file), that file will be served. DirectoryIndex index.php index.html 5. Virtual Host Setup We’ve now set up Apache to support PHP and virtual hosts. We still need to add configuration for each virtual host separately. Earlier, we uncommented a line in order to include the file ( on an Intel-based Mac). Now, let’s edit it to actually configure a virtual host. /opt/homebrew/etc/httpd/extra/httpd-vhosts.conf /usr/local/etc/httpd/extra/httpd-vhosts.conf <VirtualHost *:80>
    ServerName my-project.test
    DocumentRoot /path/to/my-project

    <Directory /path/to/my-project>
        Require all granted
        AllowOverride All
    </Directory>
</VirtualHost> This configures a virtual host on port 80. sets the hostname for the virtual site. When we visit in the browser, the change we made to will make sure the request is sent to the local Apache server. Apache will find the virtual host with the corresponding hostname and serve that site. The directive specifies the location on the file system where the files to be served exist. Make sure to change this path, along with the other instance of it in the directive, to the actual path of your website on your machine! ServerName my-project.test /etc/hosts DocumentRoot Directory The directive has to do with permissions. In the file, there’s a section which denies access to any resource by default. Access to anything that should be public has to be specifically allowed. So, for the files in our site root, we give everyone access to all resources through the directive. The directive controls which directives can be overridden in a file. A file can be used for per-directory configuration — many CMSes use this file together with the module mentioned earlier to set up their URLs, for example. Here, we allow all directives to be overridden. Directory httpd.conf Require AllowOverride .htaccess .htaccess mod_rewrite While we’re at it, we can go ahead and remove or comment out the existing dummy virtual hosts existing in this file, otherwise Apache will emit some warnings on startup, as the document roots for these sites likely don’t exist. 6. Up and Running That’s all the configuration done! All that remains is to start the server. If you want Apache to start automatically with your computer, you can use Homebrews’ command to start the server and enable it for autostart at the same time: services sudo brew services start httpd will then both turn off the server and disable autostart for it. sudo brew services stop httpd Personally, I tend to just use Apache’s built-in “server control interface” to start the server as I need it: will start the server and will stop it. sudo apachectl start sudo apachectl stop Hopefully, after starting the server, visiting your chosen hostname in your browser will result in your site being served correctly! That’s about all for today! I hope you found this guide useful. Of course, we’ve only covered Apache and PHP — there are many additions you may want to make to this stack, such as the . If you’re interested in a future article covering this or any other aspect of an Apache-based hosting stack, please ! MySQL database server let us know Also Published Here

Setting Up Apache and PHP from Homebrew on macOS

Q1 - Tell us something about your background(s). I am a distributed database engineer by heart and have been in the database space for over a decade, focusing on building highly reliable distributed systems and databases. I started my journey with Apache Cassandra back in 2012 with version 0.7 with its thrift-based Hector client and I got to spend quite a reasonable amount of time with Nate McCall. I have come to Apache Cassandra and distributed databases ecosystem from relational databases world, so it is a pretty different mindset and experience building a different kind of monolith and microservices at webscale. I joined Netflix in 2014 to help build the company’s open-source Apache Cassandra ‘muscle’ and provide expertise in microservices with scalable persistence layers. Today, the company uses Apache Cassandra heavily to satisfy its ever-growing persistence needs. Q2 - Imagine if you accidentally walked into a FinTech meet and had to explain Apache Cassandra to someone there. How would you do that? I would say Apache Cassandra is high-frequency trading (HFT) ready in terms of availability, scale, and performance. You never have to think about planned downtimes for upgrades, rollouts, and unplanned outages; you just have to plan your consistency needs appropriately. Q3 - Apache Cassandra is ‘Highly Available, Eventually Consistent database.’ What do you mean by that in terms regular joes can understand? Well, to put it simply, your database is there when you need it most and answers the questions with the same accuracy every time. Cassandra has no single point of failure as it automatically replicates to multiple nodes across data centers, making it highly available. The trade-off is some latencies in read to write performance, which means data is eventually consistent. However, you can tweak it to operate as tunable consistent for specific reads and writes. Q4 - Why do you think that ‘Zero Copy’ streaming was not an option in the earlier days? Yes, Zero Copy streaming for faster scaling operations is a great feature. When I heard about streaming in Apache Cassandra 4.0, the first thing that came to my mind was, “Why did we not do it in the first place?” This feature certainly makes C* operations much easier and tractible for large-scale deployments. The ability for nodes to stream data between each other in their clusters via SSTables makes it a comfortable fit for Kubernetes and cloud environments. Q5 - Who’s got the most intricately designed technology stack in the world, in your opinion? Every company has a unique challenge to solve, and I find the core systems that solve these tough and unique problems always have an interesting way of approaching them. I can’t speak to the approach by other companies, but at Netflix, we question our architectural assumptions and dependencies constantly. Most recently, we’ve discussed our approach to distributed tracing infrastructure, and you can read about that here . Questioning our assumptions ensures that we are always on top of our technology stack’s performance, efficiency, and scalability. One of the recent outcomes of such questioning was to raise the level of abstractions for our developers (and you can read about the on the Netflix TechBlog). Bulldozer self-serve data platform We have decades of experience operating high-performance and scalable datastores, such as Apache Cassandra. This knowledge has enabled us to build higher-level abstractions on top of data store compositions, which has resulted in increased developer velocity and optimized data store access patterns. Q6 - At Hacker Noon, we’re flooded with story submissions that claim to have the ability to build Netflix/VoD clones. Don your most cynical hat and tell us what you think? Sorry, but I’m not a cynical kind of person; I see it as a positive sign that there is a huge interest in the community to solve this problem. It is encouraging for me to see this kind of response. There are several famous people who’ve said it in slightly different ways, but without competition, where you have a monopoly, you can become complacent and settle for mediocrity and have little incentive to progress. Q7 - In closing, what would you like to tell the Hacker Noon readers, some of whom are currently using Apache tools? The Cassandra community is hosting a global party on Wednesday, April 28 to celebrate the upcoming 4.0 release milestone. It’s a one-day virtual event with three sessions that are one hour each so you can attend in your time zone. You can register for the event . here If you use or contribute to Cassandra, you can submit a lightning talk here: . https://sessionize.com/cassandra We look forward to meeting current and new users around the globe.

Here's Why Big Tech Bets Big on Apache Cassandra - An Interview With Vinay Chella of Netflix

At some point, your business is going to need to run a full LAMP (Linux Apache MySQL PHP) stack. It might be for a website, a web application, a backend, a frontend, or anything in between. If you're a developer, you certainly understand the importance of getting PHP to work with web and database servers. And if you work with one of the many PHP development services on the market, this either is or will be your bread and butter. And although you might be a rockstar it doesn't necessarily mean you know how to install PHP, MySQL, and Apache and get them configured and working with one another. PHP developer, Fortunately, help isn't far away. In fact, that's exactly what we're going to do now—install PHP, Apache, and MySQL and then find out how to configure PHP. We'll take a look at this process on the Linux operating system (both Ubuntu- and Red Hat-based platforms). If Linux isn't your tool of choice, you only need to modify a few bits of the process to make it work with either macOS or Windows. And with that said, let's get down to work. Installing PHP, Apache, and MySQL There are a number of ways to get this to work. You can install the pieces individually, or as a group (on Ubuntu). First, let's find out how to do this, piece by piece. First on Ubuntu. Remember, we want to install PHP such that it'll work with both Apache and MySQL. Log into your Ubuntu server and install Apache with the command: sudo apt-get install apache2 -y When that installation completes, install the MySQL server with the command: sudo apt-get install mysql-server -y Both of the above commands will also pick up a few other dependencies along the way. When that completes, make sure to secure the database server with the command: sudo mysql_secure_installation Finally, install PHP and the necessary bits to connect it to both Apache and MySQL with the command: sudo apt-get install php libapache2-mod-php php-mysql -y You now have Apache, MySQL, and PHP installed on Ubuntu Server, along with the software necessary for them to work together. Of course, you can make that a lot easier, by installing them all together with a single command: sudo apt-get install lamp-server^ -y Once everything is installed, you want to make sure both Apache and MySQL are started and enabled with the commands: sudo systemctl start apache2
sudo systemctl enable apache2
sudo systemctl start mysql
sudo systemctl enable mysql To do this on a Red Hat based distribution (such as CentOS 8 Stream), the commands are quite different. To install Apache, the command is: sudo dnf install httpd -y To install the MySQL database server, the command is: sudo dnf install mysql-server -y Secure the database installation with the command: sudo mysql_secure_installation Enable both Apache and MySQL with the commands: sudo systemctl start httpd
sudo systemctl enable httpd
sudo systemctl start mysql
sudo systemctl enable mysql Finally, to install PHP (and the connecting bits), you must first enable a specific repository (the EPEL repo) with the command: sudo dnf install https: //dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm Next, add the Remi repository with the command: sudo dnf install https: //rpms.remirepo.net/enterprise/remi-release-8.rpm If you get a certificate error after running this command, you might need to temporarily disable certificate checks. To do that, issue the command: sudo nano /etc/yum.conf At the bottom of that file add the line: sslverify= false Save and close the file and then re-run the above command to add the Remi repository. Finally issue the following command to install PHP and the various components: sudo dnf -y install php php-{cli,fpm,mysqlnd,zip,devel,gd,mbstring,curl,xml,pear,bcmath,json} After installing PHP, make sure to remove the sslverify=false line from the yum.conf file. Unfortunately, CentOS doesn't have an equivalent to installing lamp-server^, so you must install them one piece at a time. Configuring PHP No matter what operating system you use, PHP is configured in the php.ini file. The trick is knowing where that file is located. For example, on CentOS 8 Stream, that file is in /etc. To edit the php.ini file in CentOS 8 Stream, you'd issue the command: sudo nano /etc/php.ini On Ubuntu-based servers, the php.ini file is found in /etc/php/X/apache2/ (where X is the PHP release number). For example, on Ubuntu 20.04, you'd open the file for editing with the command: sudo nano /etc/php/ /apache2/php.ini 7.4 In Windows, that path would be C:\Program Files\PHP\X\php.ini (Where X is the release number). In macOS, that path would be /etc/php.ini. The php.ini file is quite large. As of PHP 7.4, the file is 1673 lines long, so there's a lot of configuration options to look through. However, the configuration options you'll want to edit will depend on what your needs are. However, there are a few options that are more commonly changed within the php.ini file. Those options are (listed with their default values): memory_limit = M - the maximum amount memory a script may consume.
session.gc_maxlifetime = - how much time a user can be idle before a session is expired.
upload_max_filesize = M - the maximum size the file that can be uploaded to the webserver.
post_max_size = M - the maximum value that the post variable should hold.
allow_url_fopen = On -  assists opening url objects (such files).
max_execution_time = the maximum time a script can run before being terminated ( seconds). 128 of 1440 2 of 8 in as 30 in The above list consists of common settings used with PHP in conjunction with Apache and MySQL. You might find, however, that there are other settings you need to alter to make things work a bit more smoothly. That will depend on your needs and the software you have installed on your LAMP stack. Once you have everything in place, you can test PHP by creating an info.php file with the command: sudo nano / /www/html/info.php var In that file, paste the following contents: <?php
phpinfo();
?> Save and close the file. Point a web browser to http://SERVER/info.php (Where SERVER is the IP address or the domain of the hosting server). You should then see a page listing out all of the information about your PHP installation. Conclusion And there you have it, the installation and configuration of PHP for Apache and MySQL. Once you have this stack up and running, it's ready for you to hand it over to the PHP developers in your company or those you've hired from a third-party development firm to work their magic.

How to Install and Configure PHP for Apache and MySQL

Cómo instalar y configurar PHP para Apache y MySQL

Data modeling is probably one of the most important and potentially challenging aspects of Cassandra. CQL will look familiar if you come from a relational background, but the way you use it can be very different.  For our third guide, we will walk you through the process of creating a basic data model. Read part one on Cassandra essentials and part two on bootstrapping. Keyspace Structure The first thing to learn about data modeling in Apache Cassandra is keyspaces. A is very similar to a database or schema from the relational database world, as it holds all of your tables. A keyspace also has another function; it allows you to set the replication strategy and factor for the tables it contains (we will get into that more later). Each table is organized into rows and columns. keyspace Groups of related rows -- called partitions -- are stored together on the same node(s). Logically, if you were to perform a SELECT query against a Cassandra table, it would feel pretty much the same as it does when using a relational database: you will see rows and columns in a tabular format. The difference with Cassandra is that each row has a : one or more columns that are hashed to determine which node(s) store that data. partition key Our data is “partitioned”, or subdivided, based on our partition key, which is assigned when we create our table. When we insert data, Cassandra looks at the partition key to determine where this data should go. The value of that partition key is run through a consistent hashing function and, depending on the output, we can figure out which bucket, or which range of hashes, that value fits into along the token range -- and thus, which node(s) in the cluster this data belongs on. Example users Keyspace Supposing we have a keyspace, containing a table . This table name suggests that the user data for this table will be distributed among nodes (or partitioned) by city. In this case, will be the partition key. users users_by_city city A user’s will determine where their data is stored, and users from the same city will have their data stored at the same location. Notice that we have two partitions: Phoenix and Seattle. Those partitions contain all of my rows. city In this example, represents the partition key column. But there is also the concept of as well. Clustering columns denote order, and they also contribute to the distinctiveness of the primary key, used to uniquely identify a row in a table.  Cassandra finds out the specific node (and it's replicas) responsible for that specific partition, clustering columns help in further narrowing your query search. city clustering columns Clustering columns are composed of row data in sorted order. We have two clustering columns in this example, and . Since and contain text fields, they will be naturally ordered automatically in Cassandra (if denoted a clustering column). Last Name First Name Last Name First Name Therefore, with a given partition, column values have been written to the table in ascending order (A-Z). This makes it efficient for storing and retrieving data using a clustering key. Because the is also a clustering column following , if there were multiple people with the last name “Hellson” for example, the first name would be ordered (A-Z), per the last name. Anything that is not a partition key or clustering column is just a basic data column. Last Name First Name Last Name In a relational database, even if you have indexes, you’ll end up paying for ordering on the read. Alternatively, with Cassandra, we end up paying for it on the write, as we are pre-baking the ordering of the data as it is written to the table. Therefore, when the data is written to memory and eventually flushed to disk, it is already in its ordered format. This increases the speed of data retrieval; we are optimizing for read operations. Creating a Schema in CQL Let’s take a closer look at the CQL syntax involved in creating a keyspace and table. Here we have the CQL command used to create the keyspace, including the replication strategy and the replication factor. The replication strategy tells Cassandra how to distribute the replicated data across the database machines, while the replication factor tells Cassandra how many copies of the data to create. users If you have completed the previous exercises from these articles, you are probably already familiar with CQL syntax, and the concept of replication factor. Creating tables with Cassandra is very similar to how it is done in the relational world. The big difference is that in Cassandra, we have what is called a partition key, which is always the first value of your primary key, and it is the very base unit of access when querying the database. Since it determines your data locality, it is a requirement to set one when creating your table. The values that follow the partition key within the primary key definition are the clustering columns, which we discussed earlier. So we are partitioning by , and then ordering by , and then . city last_name first_name Start with Application Workflows We've gone through the basic structure of things like keyspaces, tables, and partitions, all of which are fundamental concepts with regard to Cassandra data modeling. If you come from a relational database background, this next part is going to feel quite a bit different from what you are used to. Let’s discuss data modeling in the relational world for just a moment. Basically we start with a set of data, apply normal forms, and come up with a data model that is optimized to reduce the amount of data redundancy. As an application developer, . When we want to ask questions of our database, we’ll probably need to go look at the ERD, join some tables, and then develop our query. we develop our queries based on the table structures that have been given to us In Cassandra, the process is completely reversed. A developer with the application workflows to then develop pseudo-queries that will actually be used to generate your data model. Put another way, . This might seem like it could be a lot of work upfront, . starts you need to know your queries in advance it really just involves a shift in your thinking Once you’ve started to napkin out the UX of an app, you have already started to define your application workflows, and therefore the queries that you’re going to need to create your data model. So the very first thing we will be considering before we start data modeling is our . application workflows Unlike relational databases, Cassandra uses a denormalized data model. When using a relational database a lot of operations have to happen when you're joining tables. While that works wonderfully for a general purpose database, you may also have experienced running queries that have brought such a database down. Because of all those extra operations (Cartesian joins, all that extra IO), the database ends up performing terribly. Cassandra optimizes for read performance at scale, and that is why we use a denormalized data model. Breakdown of the Cassandra Data Modeling Process Now that you have a high-level understanding of how this process works, we are going to have a look at more of the details of data modeling with Cassandra. First, we’ll start with the conceptual data model; our entities and relationships. The process of conceptual data modeling is actually the same for both Cassandra and relational databases. The difference comes in when we bring in our application workflow to generate our queries (Remember, that is what we look at first). From there, we will generate all the way down to our physical data model and create the tables that we need. The entity-relationship diagram (ERD) below may look familiar to you. The particular relationship we're looking at comes from one of the reference that we built, KillrVideo, which is similar to YouTube. It manages videos, users, comments and so on. The diagram shows the relationship between these pieces. applications This example explores the relationships between the data points. For instance, I can look at my profile and see all the comments I've made on all the videos that I have in the system. As such,  what we want to do is tease out this relationship between , as well as , in our application workflow. To start, we would find comments that are from a particular user, and for a particular video. users and comments videos and comments Every application has a workflow: what do users see when they first come to the application? Where can the users go from there? What else can they do as they dive further into the application? refer to the specific queries that must we run to satisfy what the user is doing with our application. When a user moves from one part of an application to another part of the application, what query must we run to retrieve the data that the user is interested in seeing? Access patterns Access Pattern Examples: I’ve navigated to a video, I want to see those comments for a particular video. Since they are comments, they are also time-based, so I also want to order such that I get the most recent first. Request 1: Find comments related to target video using its identifier, also: Ensure the most the recent comment for that video is first when results are returned Implement paging such that only x number of comments are returned at once As a user, I want to long into the system, and I want to see all of my comments. I also want to see those based on time, so I get the most recent first. Request 2: Find comments related to target user using its identifier, also: Ensure the most the most recent comment from that user is first when results are returned Implement paging such that only x number of comments are returned at once Users will perform normal CRUD operations, which is something we do with all databases. They will create new accounts, update passwords, delete comments, and select videos to view. Request 3: Implement CRUD operations (Create, Read, Update, Delete) Now that we know the requests we will generate a pseudo-query for each: Let’s develop a pseudo-query for our first request. We want to find the comments for a video with some known id ( ) . Supposing you are navigating through your app, and you’ve come to a particular video of your choosing. This video has an associated identifier ( ). You are interested in viewing the comments for that particular video. This means that we should have a table (comments partitioned by video), hence comments will be queryable on the basis of the videoid associated with that particular video. Query 1: videoid videoid comments_by_videos For our second pseudo-query, we want to find the comments posted by a user with a known id ( ). If you have already logged into the app, your user account already has an associated identifier ( ). You just want to see the comments associated with your user account ( ), because you want to know what sort of comments you’ve been making on videos. Therefore, we know we would want a table (comments partitioned by user). Query 2: userid userid userid comments_by_user For both of these queries, we are going to want to get the most recent comments first. This implies that there is a somewhere in the mix. We will talk about that soon. clustering column is when we take our pseudo-queries and generate tables that would satisfy these queries. This helps us start to get a feel of what our final tables will look like. So now we have a and table, but what do these need to look like to support the queries we came up with in the mapping section? What we are looking at here is a , the Cassandra community uses these for logical data modeling. To make use of these diagrams, you need to understand the notation: Logical data modeling comments_by_user comments_by_video Chebotko diagram K - partition key C ↓ - Clustering column (descending order) C ↑ - Clustering column (ascending order) Partition keys and clustering columns are critical to get correct if we want to satisfy our queries. Let’s have a look at the columns in the table. The is the partition key here, as this table is partitioned by “user”. When we query the table, we will provide the user ( ) for which we want the comments of. Then we have the columns which pertain to the comment information itself; (in descending order, so we get the most recent on top), , (provides a link back to the video the comment is on, recall this was one of the relationships we wanted to tease out of our data points), and then the itself. comments_by_users userid userid creationdate commentid videoid comment The table has a very similar situation. The only difference of course is instead of partitioning by user, we are going to partition by video, hence is the partition key. comments_by_video videoid Notice that both tables have a lot of the same columns. We are indeed going to be repeating the same data in different tables. This is what we mean when we say Cassandra data is denormalized, for read-on-read performance, by adding redundant copies of the same data. Cassandra makes a trade-off to use more disk (which is quite cheap, nowadays) to be able to maintain the benefit of performance at scale. Now we move on from a logical data model to a model. This is when we add the data types and consider any optimizations we can make to our data model. For example, in Cassandra there is a data type called a . This is essentially a combination of a and a . So your initial inclination might be to use a for and a UUID for . physical data TIMEUUID Timestamp UUID Timestamp creationdate commentid But why not remove the column, and get the date/time associated with the creation of a comment by using a for the column. Notice how it only took a couple of steps to get from our pseudo-queries to the actual tables we were looking for. creationdate TIMEUUID commentid We can now go ahead and execute the CQL that will create these tables in our database. This is going to look very familiar if you’ve ever done this in SQL. Take note of the primary key here in the table, We are partitioning by , and ordering within that partition by ( ). This is why we set as a clustering column. The same goes for the tables. You’ve arrived, meet your data model. comments_by_users userid commitid TIMEUUID commentid comments_by_video Querying these tables is very simple. If you want to perform a basic SELECT query against the table, all you need is the partition key, . To SELECT from the table, all you need is a , the partition key for that table. Using the partition key allows us to go to the exact location of that data, which optimizes for read performance and allows queries to be fairly simple. With Cassandra, you will generally have one query per table. comments_by_user userid comments_by_videos videoid Exercise 4 - Data Modeling Our explained CQL and included an exercise. Similarly here, get the notebook in this folder and upload into a running instance of DataStax Studio and follow the instructions in the notebook. It will take you through how tables and queries in Cassandra differ from those of a relational database and enable you to start data modeling with Cassandra on your own. previous guide 4_-_Data_Modelling.tar notebooks Now you know the basics of how data modeling works in Cassandra, and the exercise walked you through the process of creating your own data model. In the next guide, we’ll explore some useful advanced data types in Cassandra, such as collections, counters, and user-defined types.

Apache Cassandra Guide: Data Modeling 

Our first article covered : the “what” and “why” about the architecture. If you’re ready to try Cassandra, the next few articles will explain the “how.” At the end of these tutorials you should have a database up and running, experienced using Cassandra Query Language (CQL), gained some practical modeling expertise, and learned best practices for building applications with Cassandra. Cassandra essentials data In the bootstrapping phase, we’re going to create a database. The next tutorial will explain how to work on that , then go deep on data modeling, after that have a look at some more advanced concepts, and then finally application development with Cassandra. database Bootstrapping Firstly, you’ll need to get your hands on the GitHub repository, where all the materials and resources are. You can either clone the repository with the following command, or download as a ZIP file (best alternative for Windows users). git ://github.com/DataStax-Academy/cassandra-workshop-online clone https Some exercises in later articles will require an IDE; you can choose your favourite one. Exercise 1 - Create an Astra instance As an application developer, unless you have a penchant for databases, you’ve probably spent a lot of time hooking up to an existing database or downloading one and installing it. DataStax Astra is an easy way to spin up a Cassandra database and use the pre-configured DataStax Studio to talk to your database and immediately get started with running CQL queries. There’s a free tier on Astra you can use to experiment, and you can connect to it using the same DataStax driver whether you’re using open source Cassandra, Astra or DataStax Enterprise. Once you are done experimenting, you can move to either your own OSS or DataStax Enterprise installation. Shifting your driver from connecting to Astra to your installation only takes a change of about 1-2 lines of code. Apache Access to the Astra service here: https://astra.datastax.com Register and Login The link above will bring you to the Astra Sign-in page. Provide your email and password, and click the Sign In button - or Sign up if you have yet to get yourself an Astra account. Fill the Create New Database Form As you don't have any instances yet, the login will route through the instance creation form. You will find below which values to enter for each field. For the work we are doing please use Free tier. Your instance will be there forever, free of charge. Set the Compute Size: This is the region where your database will reside physically (choose one close to you or your users).The idea here is to reduce latency. Select the region: - . While Astra allows you to fill in these fields with values of your own choosing, please use these suggested values to make the rest of the exercises easier to follow. If you don't, you are on your own! Fill in the database name killrvideocluster - . It's really important that you use the name killrvideo (with no 'e' in "killr") here in order for the exercises to work. You can get creative later, just roll with this for the time being. Fill in the keyspace name killrvideo - . Note the username is case-sensitive. Please use the case suggested here. Fill in the Database username KVUser - . Fill in both the password and the confirmation fields. Note that the password is also case-sensitive. Stick to the case suggested here. Fill in the Database user password KVPassword Launch the database - Review all the fields to make sure they are as shown, and click the button. Launch Database View your Database and connect Click the View Database button on the pop-up to watch your database initialize. It may take 2-3 minutes for your database to spin up. You will receive an email at that point. But go ahead and continue with the rest of the exercise now. Once the database is ready you should see the following in the Cloud Console (Click at the top left to get back to an overview of your databases). Astra Databases Let’s review the database you have configured. If you are not viewing your database already, click into the “killrvideo” database in your console. In the box on the top-left side of the window, you can see the database and keyspace name metadata. The box on the top-right describes the size and location of your database. The lower-left box shows your estimated cost ($0.00 since you are using the free tier). Once Astra initializes the database completely, the lower-right box will have connection details. Open DataStax Studio Click on the link in the Developer Studio section, in the lower left side of the console.  This should start up the Developer Studio tool. You will need to enter the username and password you used for the Astra instance creation. Launch Now - . Note the username is case-sensitive, so use the case suggested here. Fill in the Database username KVUser - . Fill in the password. The password is also case-sensitive. Supply it in the exact same format that it has been provided to you. Fill in the Database password KVPassword You should get a message welcoming you to DataStax Studio. Congratulations on setting up your first Cassandra database with Astra! Wasn’t that easy? https://ctt.ac/ZaaFT Exercise 2 - Launch DataStax Studio DataStax Studio is a great developer tool that we are going to use for our exercises. It is automatically configured to talk to our Astra database. Now from your cloned repo, you are going to drag-and-drop a Studio Notebook into your Studio instance to begin your next exercise. cassandra-workshop-online Grab the file from the “notebooks” folder and drop it in. Alternatively, you can upload it from the hamburger menu in the upper left hand corner of the console. Once you have the notebook up, you can begin following the instructions, including executing a CQL script to initialize the killrvideo database for this notebook. 2_-_Datastax_Studio.tar We use the cells in Studio for the various steps of the exercises. Some cells are Markdown cells, which contain formatted text. Markdown is used for adding text or instructions to an exercise step in the notebook. Other cells are CQL cells, and allow you to execute CQL queries against your database directly from the notebook. You can work through this notebook at your own pace, and be sure to head over to our if you have questions! Community forum Exercise 3 - Working with CQL CQL (Cassandra Query Language) is a structured query language used to access Cassandra databases -- like SQL for relational databases. In this notebook, try out CQL and familiarize yourself with the language. If you already know SQL, CQL will be very intuitive. To get started on the next exercise, use the hamburger menu to get back to where your notebooks are located in Studio. Upload the notebook into your running instance of DataStax Studio: 3_-_Working_with_CQL.tar Once again, just follow the instructions in the notebook. It will take you through learning the CQL syntax to enable you to write statements to add and manipulate data in your Astra database. You did it! That’s it for this tutorial. The next one will cover data modeling in Cassandra.

Apache Cassandra – Bootstrapping

When it comes to Big Data infrastructure on , the most popular choices Data architects need to consider today are – A serverless, highly scalable and cost-effective cloud data warehouse, Apache Beam based Cloud Dataflow and – a fully managed cloud service for running and clusters in a simpler, more cost-efficient way. Google Cloud Platform Google BigQuery Dataproc Apache Spark Apache Hadoop This variety also presents challenges to architects and engineers looking at moving to Google Cloud Platform in selecting the best technology stack based on their requirements and to process large volumes of data in a cost effective yet reliable manner. In the following sections, we look at research we had undertaken to provide interactive business intelligence reports and visualisations for thousands of end users. Furthermore, as these users can concurrently generate a variety of such interactive reports, we need to design a system that can analyse billions of data points in real time. Requirements For technology evaluation purposes, we narrowed down to following requirements – Raw data set of size : This dataset is quite diverse with scores of tables and columns consisting of metrics and dimensions derived from multiple sources. 175TB Catering to 30,000 unique users Serving up to to the platform users 60 concurrent queries The problem statement due to the size of the base dataset and requirement for a high real time querying paradigm requires a solution in the Big Data domain. Salient Features of Proposed Solution The solution took into consideration following 3 main characteristics of desired system: Analysing and classifying expected user queries and their frequency. Developing various pre-aggregations and projections to reduce data churn while serving various classes of user queries. Developing state of the art ‘Query Rewrite Algorithm’ to serve the user queries using a combination of aggregated datasets. This will allow the Query Engine to serve maximum user queries with minimum number of aggregations. Tech Stack Considerations For benchmarking performance and the resulting cost implications, following technology stack on Google Cloud Platform were considered: 1. Cloud DataProc + Google Cloud Storage For Distributed processing – Apache Spark on Cloud DataProc For Distributed Storage – Apache Parquet File format stored in Google Cloud Storage 2. Cloud DataProc + Google BigQuery using Storage API For Distributed processing – Apache Spark on Cloud DataProc For Distributed Storage – BigQuery Native Storage (Capacitor File Format over Colossus Storage) accessible through BigQuery Storage API 3. Native Google BigQuery for both Storage and processing – On Demand Queries Using BigQuery Native Storage (Capacitor File Format over Colossus Storage) and execution on BigQuery Native MPP (Dremel Query Engine) All the queries were run in on demand fashion. Project will be billed on the total amount of data processed by user queries. 4. Native Google BigQuery with fixed price model Using BigQuery Native Storage (Capacitor File Format over Colossus Storage) and execution on BigQuery Native MPP (Dremel Query Engine) Slots reservations were made and slots assignments were done to dedicated GCP projects. All the queries and their processing will be done on the fixed number of BigQuery Slots assigned to the project. Tech Stack Performance Comparison After analyzing the dataset and expected query patterns, a data schema was modeled. Dataset was segregated into various tables based on various facets. Several layers of aggregation tables were planned to speed up the user queries. All the user data was partitioned in time series fashion and loaded into respective fact tables. Furthermore, various aggregation tables were created on top of these tables. All the metrics in these aggregation tables were grouped by frequently queried dimensions. In the next layer on top of this base dataset various aggregation tables were added, where the metrics data was rolled up on a per day basis. All the probable user queries were divided into 5 categories Raw data and lifting over 3 months of data Aggregated data and lifting over 3 months of data Aggregated data over 7 days. Aggregated data over 15 days. Raw data over 1 month. The total data processed by individual query depends upon time window being queried and granularity of the tables being hit. Query Response times for large data sets – Spark and BigQuery Test Configuration Total Threads = 60,Test Duration = 1 hour, Cache OFF 1) Apache Spark cluster on Cloud DataProc Total Nodes = 150 (20 cores and 72 GB), Total Executors = 1200 2) BigQuery cluster BigQuery Slots Used = 1800 to 1900 Query Response times for aggregated data sets – Spark and BigQuery Test Configuration Total Threads = 60,Test Duration = 1 hour, Cache OFF 1) Apache Spark cluster on Cloud DataProc Total Machines = 250 to 300, Total Executors = 2000 to 2400, 1 Machine = 20 Cores, 72GB 2) BigQuery cluster BigQuery Slots Used: 2000 Performance testing on 7 days data – Big Query native & Spark BQ Connector It can be seen that BigQuery Native has a processing time that is ~1/10 compared to Spark + BQ options Performance testing on 15 days data – Big Query native & Spark BQ Connector It can be seen that BigQuery Native has a processing time that is ~1/25 compared to Spark + BQ options Processing time seems to reduce with increase in the data volume Longevity Tests – BigQuery Native REST API Once it was established that BigQuery Native outperformed other tech stack options in all aspects. We also ran extensive longevity tests to evaluate response time consistency of data queries on BigQuery Native REST API. It is evident from the above graph that over long periods of running the queries, the query response time remains consistent and the system performance and responsiveness doesn’t degrade over time. ETL performance – BigQuery Native To evaluate the ETL performance and infer various metrics with respect to execution of ETL jobs, we ran several types of jobs at varied concurrency. In BigQuery, similar to interactive queries, the ETL jobs running in batch mode were very performant and finished within expected time windows. This should allow all the ETL jobs to load hourly data into user facing tables and complete in a timely fashion. Running the ETL jobs in batch mode has another benefit. All jobs running in batch mode do not count against the maximum number of allowed concurrent BigQuery jobs per project. Comparing costs – BigQuery & Spark Here we capture the comparison undertaken to evaluate the cost viability of the identified technology stacks. Two Months billable dataset size in BigQuery: 59.73 TB. Two Months billable dataset size of Parquet stored in Google Cloud Actual Data Size used in exploration: Storage: 3.5 TB. Parquet file format follows columnar storage resulting in great compression, reducing the overall storage costs. Actual Data Size used in exploration: BigQuery 2 Months Size (Table): 59.73 TB Spark 2 Months Size (Parquet): 3.5 TB In BigQuery – storage pricing is based on the amount of data stored in your tables when it is uncompressed In BigQuery – even though on disk data is stored in Capacitor, a columnar file format, storage pricing is based on the amount of data stored in your tables when it is uncompressed. Hence, Data Storage size in BigQuery is higher than that in Spark on GCS in parquet format. ~17x Conclusion For both small and large datasets, user queries’ performance on BigQuery Native platform was significantly better than that on Spark Dataproc cluster. Query cost for both On Demand queries with BigQuery and Spark based queries on Cloud DataProc is substantially high. Using BigQuery with Flat-rate priced model resulted in sufficient cost reduction with minimal performance degradation. About the Authors Prateek Srivastava is Technical Lead at with expertise in BigData, Streaming, Cloud and Service Oriented architecture. Roushan is a Software Engineer at , who works on building ETL pipelines and Query Engine on  Apache Spark & BigQuery, and optimising query performance Sigmoid Sigmoid Previously published at https://www.sigmoid.com/blogs/apache-spark-on-dataproc-vs-google-bigquery/

Performance Benchmark: Apache Spark on DataProc Vs. Google BigQuery

he web server has a crucial role in web-based applications. Since most of us leave it to the default configuration, it can leak sensitive data regarding the web server. There are numerous web servers in the market. Apache is one of the most popular and widely used out of all of them. Because of this popularity, it is also most vulnerable to cyber-attacks. By applying numerous configuration tweaks we can make Apache withstand malicious attacks up to a limit. Following are some Apache web server hardening tips that you can incorporate to improve security. Hiding Server Version Banner One of the first things to be taken care of is hiding the server version banner. The default Apache configuration will expose the server version. This information might help an attacker gain a greater understanding of the systems in use and potentially develop further attacks targeted at the specific version of the server. We can easily fix server version disclosure by following the below steps: 1. Open apache.conf # vim /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora)
# vim /etc/apache2/apache2.conf (Debian/Ubuntu)
Add the following directives to configuration
```
ServerSignature Off
ServerTokens Prod
``` 2. Save the configuration and restart Apache Even better, we can change the server name to anything else in the server header. In order to achieve this, you need to enable the mod_security module. Then add the following directives to the configuration. ServerSignature Off
        ServerTokens Prod 3. Save the configuration and restart Apache Disable Directory Listing By Apache’s default configuration, If your web server root directory doesn’t contain index.html, the user can see all files and sub directories listed in the web root. To disable directory listing we need to set the `Option` directive value as `None` or `-Indexes` in the Apache configuration file. Example: < Directory /var/www/html>
    Options -Indexes
</Directory> Restart apache2 Use mod_security Module mod_security works as a firewall for web applications. It can also be used for real-time web application monitoring and logging. You can install mod_security from your default package installer. Installation Debian/Ubuntu # apt install libapache2-mod-security2
# service apache2 restart Installation RHEL/CentOS/Fedora # yum install mod_security # systemctl restart httpd.service Use mod_evasive Module mod_evasive provides effective actions against Distributed Denial of Service (DDoS/DoS) attack or brute force attack Its capabilities also extend to work with ipchains, firewalls, routers, and more. mod_evasive reports events via email and syslog facilities. mod_evasive has a prerequisite. Install the prerequisite by running the following command. Prerequisites Debian/Ubuntu # apt install apache2-utils RHEL/CentOS/Fedora # yum install httpd-devel Installation Debian/Ubuntu # apt install libapache2-mod-evasive RHEL/CentOS/Fedora # yum install mod_evasive Configuring mod_evasive Open mod_evasive configuration file in any text editor. you can find the configuration file in the following path: Debian/Ubuntu # vim apache2 evasive.conf /etc/ /mods-enabled/ RHEL/CentOS/Fedora # /etc/httpd/ .d/mod_evasive. vim conf conf Find the following lines and uncomment them. DOSHashTableSize 3097 DOSPageCount 2 DOSSiteCount 50 DOSPageInterval 1 DOSSiteInterval 1 DOSBlockingPeriod 10 DOSEmailNotify mail@yourdomain.com DOSLogDir "/var/log/apache2/" Replace `DOSEmailNotify ` with your email address to get a notification on events. Email notifications only work if there’s a functional mail server running on the server. mail@yourdomain.com Save the configuration and restart Apache. Now, mod_evasive is in effect. Hiding ETag Header The ETag header involves quite a significant number of sensitive details regarding your server. It's interesting that for PCI-compliance it is required to hide the Etag header. To achieve this, add the following directive to Apache configuration. FileETag None Disable CGI and SSI SSIs are directives present on web applications used to feed an HTML page with dynamic contents. They are also capable of opening your website up to a certain number of security issues if left unchecked. The same case happens for the CGI scripts. So as to prevent hackers from injecting malicious scripts in your code. Restrict CGI and SSI by adding the following directives to Apache configuration: - -ExecCGI Options Includes Options Setting the HTTP Limits Setting up some HTTP limits can defend against DDoS (Distributed denial of service) attack, it is really easy if you know the sort of actions to look out for. DDoS always tends to happen by hitting repeatedly on your server with very large requests. The following include some limits you make need to configure: KeepAlive= KeepAliveTimeout
LimitRequestBody
LimitRequestFields
LimitRequestFieldSize
LimitRequestLine
LimitXMLRequestBody
MaxClients
MaxKeepAliveRequests
MaxRequestWorkers
RequestReadTimeout
TimeOut on Enable XSS Protection Header Cross-site scripting (XSS) is a common vulnerability found in web applications. X-XSS-Protection header can prevent some level of XSS (cross-site-scripting) attacks. The parameters are: 0 - XSS filter disabled 1 - XSS filter enabled and sanitizes the page if attack detected 1;mode=block - XSS filter enabled and prevents rendering the page if attack detected 1;report=http://reporting.url/ - XSS filter enabled and will report the violation if an attack detected Add the following entry to your Apache configuration to enable XSS Protection Header. Header X-XSS-Protection set "1; mode=block" Restart Apache. Last but not the least, always keep your web server updated. I hope these Apache web server security hardening tips come in handy for you! If you're looking to improve your web security, you can on Beagle Security. You'll be able to identify vulnerabilities on your website before hackers exploit them. Stay secure! sign up for a free account Previously published at https://beaglesecurity.com/blog/blogs/2020/06/15/Apache-Web-Server-Hardening.html

Apache Web Server Hardening: How To Protect Your Server From Attacks

Need to get up to speed on Cassandra and learn how it can benefit your software development practices? These are the essentials that cover the basics. Apache Cassandra is an extremely powerful NoSQL database used by Apple, CERN, GitHub, Instagram, Netflix, Reddit and over 1500 more companies with large, active data sets. Recent developments have made it much easier for developers to use, including the creation of Kubernetes Operators and “as a service” options. Before you start using Cassandra, it is good to get an overview of its capabilities, a sense of the pieces that comprise a solution, and the tools that make the integration possible. In this article, we explain Cassandra’s approach, architecture, key concepts, and primary use cases – and demonstrate the reasons why so many people want to adopt it. Let’s start with its basic concepts. Cassandra as a NoSQL database Cassandra is a NoSQL distributed database. By design, NoSQL databases are lightweight, open-source, non-relational, and largely distributed. Counted among their strengths are horizontal scalability, distributed architectures, and a flexible approach to schema definition. NoSQL databases enable rapid, ad-hoc organization and analysis of extremely high-volume, disparate data types. That’s become more important in recent years, with the advent of Big Data and the need to rapidly scale databases in the cloud. Cassandra is among the NoSQL databases that have addressed the constraints of previous data management technologies, such as the SQL databases on which many developers cut their teeth. Distribution provides power and resilience One important Cassandra attribute is that its databases are distributed. That yields both technical and business advantages. Cassandra databases easily scale when an application is under high stress, and the distribution also prevents data loss from any given datacenter’s hardware failure. A distributed architecture also brings technical power; for example, a developer can tweak the throughput of read queries or write queries in isolation. “Distributed” means that Cassandra can run on multiple machines while appearing to users as a unified whole. There is actually little point in running Cassandsra as a single node, although it is very helpful to do so to help you get up to speed on how it works. But to get the maximum benefit out of Cassandra, you would run it on multiple machines. Since it is a distributed database, Cassandra can (and usually does) have multiple nodes. A node represents a single instance of Cassandra. These nodes communicate with one another through a protocol called gossip, which is a process of computer peer-to-peer communication. Cassandra also has a masterless architecture – any node in the database can provide the  exact same functionality as any other node – contributing to Cassandra’s robustness and resilience. Multiple nodes can be organized logically into a cluster, or “ring”. You can also have multiple datacenters. Want more power? Add more nodes One reason for Cassandra’s popularity is that it enables developers to scale their databases dynamically, using off-the-shelf hardware, with no downtime. You can expand when you need to – and also shrink, if the application requirements suggest that path. Perhaps you are used to Oracle or MySQL databases. If so, you know that extending them to support more users or storage capacity requires you to add more CPU power, RAM, or faster disks. Each of those costs a significant amount of money. And yet: Eventually you still encounter some ceilings and constraints. In contrast, Cassandra makes it easy to increase the amount of data it can manage. Because it’s based on nodes, Cassandra scales horizontally (aka scale-out), using lower commodity hardware. To double your capacity or double your throughput, double the number of nodes. That’s all it takes. Need more power? Add more nodes – whether that’s 8 more or 8,000 – with no downtime. You also have the flexibility to scale back if you wish. This linear scalability applies essentially indefinitely. This capability has become one of Cassandra’s key strengths. Introducing Partitions In Cassandra, the data itself is automatically distributed, with (positive) performance consequences. It accomplishes this using partitions. Each node owns a particular set of tokens, and Cassandra distributes data based on the ranges of these tokens across the cluster. The is responsible for distributing data among nodes and is important for determining data locality. When data is inserted into the cluster, the first step is to apply a hash function to the partition key. The output is used to determine what node (based on the token range) will get the data. partition key When data comes in, the database’s coordinator takes on the job of assigning to a given partition – let’s call it partition 59. Remember that any node in the cluster can take on the role as the coordinator. As we mentioned earlier, nodes gossip to one another; during which they communicate about which node is responsible for what ranges. So in our example, the coordinator does a lookup: Which node has the token 59? When it finds the right one, it forwards that data to that node. The node that owns the data for that range is called a replica node. One piece of data can be replicated to multiple (replica) nodes, ensuring reliability and fault tolerance. So far, our data has only been replicated to one replica. This represents a replication factor of one, or RF = 1. The coordinator node isn’t a single location; the system would be fragile if it were. It’s simply the node that gets the request at that particular moment. Any node can act as the coordinator. Replication ensures reliability and fault tolerance One piece of data can be replicated to multiple (replica) nodes, ensuring reliability and fault tolerance. Cassandra supports the notion of a (RF), which describes how many copies of your data should exist in the database. So far, our data has only been replicated to one replica (RF = 1). If we up this to a replication factor of two (RF = 2), the data needs to be stored on a  second replica as well – and hence each node becomes responsible for a secondary range of tokens, in addition to its primary range. A replication factor of three ensures that there are three nodes (replicas) covering that particular token range, and the data is stored on yet another one. replication factor The distributed nature Cassandra makes it more resilient and performant. This really comes into play when we have multiple replicas for the same data. Doing so helps the system to be self-healing if something goes wrong, such as if a node goes down, a hard drive fails, or AWS resets an instance. Replication ensures that data isn’t lost. If a request comes in for data, even if one of our replicas has gone down, the other two are still available to fulfill the request. The coordinator stores a “hint” for that data as well, and when the downed replica comes back up, it will find out what it missed, and catch up to speed with the other two replicas. No manual action is required, this is done completely automatically. The use of multiple replicas also has performance advantages. Because we aren’t limited to a single instance We have three nodes (replicas) that can be accessed to provide data for our  operations, which we can load balance amongst to achieve the best performance. Cassandra automatically replicates that data around your different data centers. Your application can write data to a Cassandra node on the U.S. west coast, and that data is automatically available in data centers at nodes in Asia and Europe. That has positive performance advantages – especially if you support a worldwide user base. In a world dependent on cloud computing and fast data access, no user suffers from latency due to distance. Tuning your consistency We’ve been talking a lot about distributed systems and availability. If you are familiar with CAP theorem, Cassandra is by default an AP (Available Partition-tolerant) database, hence it is “always on”.  But you can indeed configure the consistency on a per-query basis.  In this context, the consistency level represents the minimum number of Cassandra nodes that must acknowledge a read or write operation to the coordinator before the operation is considered successful. As a general rule, you will select your consistency level (CL) based on your replication factor. For the example below, our data is replicated out to three nodes. We have a CL=QUORUM (Quorum referring to majority, 2 replicas in this case or RF/2 +1) therefore the coordinator will need to get acknowledgement back from two of the replicas in order for the query to be considered a success. As with other computing tasks, it can take some skill to learn to tune this feature for ideal performance, availability, and data integrity – but the fact that you can control it with such granularity means you can control deployments in great detail. Ultimately, Cassandra is deployment agnostic. It doesn't care where you put it – on prem, a cloud provider, multiple cloud providers. You can use a combination of those for a single database. That gives software developers the maximum amount of flexibility. There’s more to learn about the fundamentals of Cassandra architecture. But a grounding in these basics will serve you well as you dive deeper into the subject.

Apache Cassandra – An Essentials Guide

A Detailed Guide To Using Apache Storm

A Comprehensive Guide to Apache Cassandra Architecture

Oi pessoal! Este tutorial mostra como instalar um servidor Web Apache. Se você já lutou sobre esse assunto, junte-se a mim! Podemos facilitar um pouco para todos os outros. Esta instalação usará os binários do Apache, não usaremos um gerenciador de pacotes, como o yum e o apt. Mas ... por que você escolheria usar binários? Você tem um gerenciador de pacotes que faz isso por você! Bem, existem 2 razões principais para isso: Talvez você queira uma versão específica do Apache, que o seu gerenciador de pacotes não suporta Talvez o seu servidor não tenha uma conexão com a Internet. Ok, vamos lá... Se você acessar o e baixar a versão desejada, provavelmente falhará durante a instalação. Isso ocorre porque o HTTPD precisa de uma variedade de bibliotecas que talvez você ainda não tenha instalado. site do Apache Você precisará de duas outras bibliotecas para obter uma instalação sem falhas: APR APR-Util APR significa Apache Portable Runtime, que é um conjunto de bibliotecas que suportam HTTPD com um conjunto de APIs que mapeiam seu sistema operacional para atender às múltiplas solicitações que chegam ao seu servidor da web. APR-Util é um conjunto de bibliotecas que complementam a APR Então, como faço isso? A ordem da instalação é esta: APR APR-Util Servidor Web Apache Você pode baixar o APR e APR-Util . aqui Você pode baixar o servidor da web Apache . aqui Todo o processo é sobre compilação e instalação. Então vamos! Antes de tudo, configure uma estrutura de diretórios para o seu servidor Web. Eu gosto de fazê-lo em . /opt cd /opt/
mkdir httpd/ Dentro da pasta que você criou, crie outros 2 diretórios: cd httpd/
mkdir apr/
mkdir apr-util/ Para melhores práticas, convém usar as versões no nome desses diretórios. Faça o download dos binários e coloque esses diretórios.
Agora, você precisa carregar essas configurações na memória. Vá para o diretório APR, este será o primeiro instalado: cd apr/
./configure --prefix=/opt/httpd/apr/
make Agora é hora do APR-Util: cd /opt/httpd/apt-util/
./configure --prefix=/opt/httpd/apr-util/ --with-apr=/opt/httpd/apr/
make
make install É basicamente a mesma coisa, exceto que você usará o APR para carregar a configuração do APR-Util Agora, vem a última parte. E o mais esperado! Você só precisa instalar o servidor da Web: cd /opt/httpd/
./configure --prefix=/opt/httpd/ --with-apr=/opt/httpd/apr/ --with-apr-util=/opt/httpd/apr-util/
make
make install Se você não encontrar nenhum problema de versão, poderá: sudo /opt/httpd/bin/httpd -k start E isso iniciará o seu servidor Web. Você precisa usar raiz para este comando, porque a porta padrão que o Apache executa é apenas utilizável pela raiz. Eu recomendo que você defina um usuário específico para o Apache, isso economizará muito tempo. Se você quiser ver qual versão do APR e APR-Util você deve usar, confira . aqui Eu realmente espero que você tenha gostado deste tutorial. É simples, mas é o meu primeiro! Se você quiser falar comigo, pode me adicionar no . Linkedin De qualquer forma, até breve! Previously published at https://medium.com/@mschirbel/how-to-install-an-apache-web-server-b8d1684a95ef

Como Instalar um Servidor Apache HTTPD

About Performance Testing Performance testing is a method of identifying whether software (a web application) is capable to handle a high load, among other possible performance issues. Therefore, the ultimate goal of a performance test is to analyze the overall performance against heavy loads. Apache JMeter is an open-source tool that can be used exactly for this purpose. What is a Single-node Server? This is the case when a server runs on a single machine only. If the server crashes due to some issue, there will be a downtime until the issue is fixed. What is a Two-node Cluster? A two**-**node cluster is a setup where two servers are configured with high availability. If clustering is used, there won’t be downtime due to a server crash. The advantage of having your servers in a cluster environment is that if one server is unavailable due to a crash or some other failure, the other node will operate as the primary one and the system will be operated without causing downtime for users. Why Apache JMeter Is the Best Tool for It Apache JMeter enables test performance for static and dynamic resources. Static resources include JavaScript, HTML, and others. Dynamic resources include JSP, Servlets, AJAX, and more. In addition, Apache JMeter has more properties that make it a suitable tool for performance and load testing: · Overall performance analysis presented as a report · A performance testing report containing analysis of different attributes, such as response time, page load time, throughput, error rate, and more · Optimization suggestions provided in a report · Graphical analysis reports with metrics such as transaction per second, hits per second, etc. · Ability to identify the maximum number of concurrent users that can be handled without the server crashing · Ability to extract data from a number of response types such as HTML, JSON, XML, etc. · Support of an extensible core with pluggable samplers, scriptable samplers, pluggable timers, and visualization plugins · Ability to integrate with any third-party open-source libraries such as Maven, Gradle, and Jenkins · Support of multithreading frameworks A performance test can be carried out in either of the two ways mentioned below: this is a testing type that models the expected usage by simulating multiple concurrent user accesses. Load Testing: this is a testing type used to identify the maximum load that can be handled by the server. When the load goes above the benchmark, the server may start responding slowly. Stress Testing: In the following, we will see how we can measure server performance using Apache Jmeter using the standard practice adopted at . Further, I will explain how this can be extended to test the performance of two servers running in parallel in a two-node cluster environment. You can use any web app that you are familiar with to conduct the test. I will be using eBay for this purpose. A1QA This process will consist of five main steps: Add a Thread Group Add necessary JMeter Elements Add Graph Results Run the test Obtain and analyze the results 1. Let’s Get a Thread Group The Thread Group element will be the initial step of the test plan. The Thread Group is a set of threads that will be executing the same scenario. This defines a pool of users participating in a particular test case. Other samplers and controllers need to be included inside the Thread Group. This article assumes that you have installed Apache JMeter on your machine. If not, please refer to for more details. this official guide How to add a Thread Group: Start Apache JMeter See the Test Plan option on the left-hand side Right-click on the Test Plan option Select the Add option Select the Threads (Users) option Select the Thread Group option After adding a Thread Group, you will see the Thread Properties section where you can control the properties of the Thread Group. : 100 (number of users connected to the web app: 100) Number of Threads ● : 10 (number of times to execute testing) Loop Count ● : 100 (time required for JMeter to get all the threads up and running. Also, this is explained as the delay time before the start of the next user.) Ramp-Up Period 2. Let’s Add Necessary JMeter Elements We will be using two elements in this use case. They are HTTP Request Default and HTTP Request. How to add an HTTP Request Default You can add an HTTP Request Default sampler in the following way: 1. Right-click on the Thread Group from the tree node 2. Select the Add option 3. Select the Config Element option 4. Select the HTTP Request Defaults option You can see input values for the server IP and Port on the control panel. This is where you will input the IP and the port relevant to the web app for which you are carrying out the performance test. In this tutorial, I will be using the IP of eBay. How to add an HTTP Request You can add an HTTP Request sampler in the following way: 1. Right-click on the Thread Group from the tree node 2. Select the Add option 3. Select the Sampler option 4. Select the HTTP Request option After adding the HTTP Request Sampler, you can see the Path field as shown in the image below. You can input a specific path to which you need to send the HTTP Request. For example, if you need to send the request to the Daily Deals section in eBay, you can check the path address relevant to that section from the website and input it under the path section: Path: /globaldeals So if you input /globaldeals for the path section, JMeter will send the HTTP Request to the complete URL . If the path field is blank, it will send the request to . https://www.ebay.com/globaldeals https://www.ebay.com I will keep the path-field blank. So, the requests will be sent to https://www.ebay.com 3. Let’s Add Graph Results You can use Graph Result Listener to observe the test results in the following way: 1. Right-click on Test Plan 2. Select Add 3. Select Listener 4. Select Graph Results 4. Let’s Run the Performance Test Before running the test, you need to save your test plan by giving a name such as Perf-test.jmx. Then click on the green mark in the toolbar to run the test. 5. Let’s Obtain and Analyze the Results While running the test, you can observe the test results which are displayed as Graphs in real time. You can see the graph that shows the performance results obtained for the sample use case we tried, in which 100 users accessed eBay server. The statistics displayed at the bottom of the graph according to their colours: ● Black shows the total number of current samples sent. ● Blue shows the average value for the total number of samples sent. ● Red shows the standard deviation. ● Green shows the throughput rate (This is responsible for the total number of requests per minute that the server handled) To gain an idea about the performance of the eBay server, we need to focus on two parameters: throughput and deviation. is the ability of the server to handle a heavy load. The higher this value, the better performance of the server. Throughput In this sample test, the throughput of the eBay server is 199.609/min. This means the eBay server can handle 199.609 requests per minute. This value is quite low compared with other web servers such as Google. Likewise, you can take the values for different servers and compare them with other servers to get an idea about the overall performance. indicates the deviation from the average. The smaller value of deviation, the better server performance. Deviation Extending the Performance Test to Two Servers Running In Parallel In a Two-node Cluster Environment Let’s assume your servers are deployed in a cluster environment where the cluster operates with two nodes. Let’s see how you can carry out a performance test against a setup similar to this: Add Thread Group Add necessary JMeter Elements Add Graph Results Run the test Obtain and analyze the results Step 1 This is similar to what I explained above. You can add a Thread Group by following the same steps. After adding the Thread Group, configure the properties as shown below. Step 2 Now, we need to add two HTTP Requests to send requests simultaneously to two nodes. There, you can follow the steps that I have explained above to add an HTTP Request. Under the Methods section, specify the method of the request. As I am sending a POST request, I have changed it for the type POST. Then, under the Path section specify the URL of the server including the IP address and the port. An example is shown below. You need to add two HTTP Requests including the same configs other than the server URLs. When two servers are running on a cluster, the URLs for the two servers will be different. Therefore, when you are adding the second HTTP Request, specify the URL relevant to the second node. Step 3, Step 4, Step 5 will be similar to the ones explained above. Therefore you can just follow the same actions. Summing Up In this article, we learned how to carry out a performance test of a web server with a single node and how to extend the test to a two-node cluster using Apache JMeter. We also discussed how we can analyze and compare the results with those of other servers.

Apache News on
HackerNoon and Around the Web

apachecorp.com

3163 employees

Since 1954

517673 at Alexa

Apache Story Mentions

Articles Around the Web

Read More Tech Stories Related to #Apache

Apache News on HackerNoon and Around the Web