Known as a transformational change agent, Campbell joins ALTR from Incorta and previously held leadership roles overseeing strategy and growth for Collibra, Aginity, MapR, VMware, and HP. He will spearhead expanding the open-source integration initiative ...

ALTR Launches Open-Source Data Governance Integration Initiative

Get the most recent info and news about Aginity on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Aginity LLC was founded in 2005. The company's line of business includes providing computer related services and consulting.

Aginity

Top Software Development Companies 2022 That Are Growing Fast

Subscribe to my Data Youtube Channel and Podcasts, Links Here Data Vault modeling is an approach to data warehouse design that offers a unique method for handling complex data from disparate sources in a way that is agile, flexible, and scalable. Developed by Dan Linstedt in the early 2000s, Data Vault modeling addresses many of the challenges associated with traditional data warehousing methods, such as the star schema and snowflake schema. By emphasizing the separation of concerns between the structure of data and the integration of data, Data Vault enables businesses to adapt quickly to changes in their data environment, making it an ideal choice for dynamic, fast-paced industries. The core philosophy of Data Vault modeling revolves around capturing data in its most granular form, ensuring that every piece of information is stored without loss of detail or context. This approach not only facilitates better data governance and compliance but also enhances data quality and usability across the organization. Core Concepts of Data Vault Modeling At the heart of Data Vault modeling are three primary concepts: Hubs, Links, and Satellites. Each plays a vital role in the structure and function of a Data Vault model. represent the unique business keys within the organization. They are the cornerstone of the Data Vault architecture, ensuring that data is accurately linked across various sources. Hubs connect Hubs together, forming relationships between different pieces of data. These Links are crucial for establishing the context and associations among disparate data elements, making it possible to construct a comprehensive view of the organization's data landscape. Links provide descriptive details about the Hubs and Links, such as attributes, historical changes, and timestamps. Satellites allow for the storage of context and change over time, enabling detailed analysis and reporting. Satellites Data Vault modeling stands out by offering a way to accommodate big data and analytics needs, thanks to its inherent scalability and flexibility. Unlike traditional models that may require significant redesign as new data sources are introduced, Data Vault's modular nature allows for easy expansion and adaptation without disrupting existing structures. Comparing Data Vault with other data modeling techniques highlights its advantages in handling complex and changing data environments. While star schemas and snowflake schemas offer simplicity and are well-suited for specific reporting needs, they lack the agility and scalability that Data Vault brings to handling diverse and evolving data sources. Benefits of Data Vault Modeling Implementing Data Vault modeling in data warehousing projects brings several key benefits: Scalability and Flexibility Due to its modular design, Data Vault can easily scale to accommodate growing data volumes and complexity. New Hubs, Links, and Satellites can be added to the model without disrupting existing structures, making it highly adaptable to changes in business requirements. Ease of Integration Data Vault's structure simplifies the integration of data from disparate sources, providing a consistent and unified view across the organization. Its design enables the efficient incorporation of new data sources, facilitating a more comprehensive data strategy. Improved Data Quality and Governance The granular approach of Data Vault modeling enhances data quality, as each piece of information is stored in detail with clear lineage. This structure also supports better data governance practices, with built-in mechanisms for compliance, auditability, and data security. Enhanced Auditability and Compliance Features Data Vault's inherent tracking of historical data and its ability to capture metadata make it exceptionally well-suited for meeting audit and compliance requirements. The model's design ensures that changes are recorded in a way that supports traceability and accountability, essential for regulatory compliance. These benefits make Data Vault modeling an attractive option for organizations looking to build robust, scalable, and compliant data warehousing solutions that can adapt to future needs and challenges. Planning and Implementing a Data Vault Model Implementing a Data Vault model in your organization requires careful planning and a structured approach. Here's a step-by-step guide to help you get started: Step-by-Step Guide on Starting with Data Vault Modeling : Begin by gathering detailed business requirements and understanding the data needs of your organization. This will guide the design of your Data Vault model. Understand Your Business Requirements : Evaluate your existing data infrastructure to determine how well it can support a Data Vault architecture. Consider factors like storage capacity, processing power, and existing data integration processes. Assess Your Current Data Infrastructure : Based on your requirements, start designing your Data Vault model. Identify the Hubs, Links, and Satellites that will form the structure of your data warehouse. Design Your Data Vault Model : Choose the tools and technologies that best support Data Vault modeling. This may include specific database platforms, ETL tools, and data modeling software. Select the Right Tools and Technologies : Begin the construction of your Data Vault by creating the Hubs, Links, and Satellites. Implement data loading processes to populate your Data Vault. Build and Deploy Your Data Vault : As your data needs evolve, continue to expand and refine your Data Vault model by adding new Hubs, Links, and Satellites as required. Iterate and Expand Key Considerations: Infrastructure, Tools, and Team Skills : Ensure your infrastructure can handle the volume and velocity of data expected. Scalability and performance are key. Infrastructure : Select tools that are compatible with Data Vault modeling principles and can automate parts of the process to improve efficiency. Tools : Your team should have a strong understanding of Data Vault principles, as well as skills in data modeling, database management, and ETL processes. Team Skills Data Vault Project Lifecycle The lifecycle of a Data Vault project encompasses several phases, from initial planning and design to deployment and maintenance. It's important to adopt an iterative approach, allowing for continuous improvement and adaptation to new business requirements. Best Practices for Data Vault Modeling Adhering to best practices in Data Vault modeling can significantly enhance the success of your data warehousing project. Here are some key tips to follow: Effective Hub, Link, and Satellite Design : Focus on business keys that are stable and uniquely identify business objects. Hubs : Model relationships clearly, including many-to-many relationships, to accurately represent business processes. Links : Capture as much descriptive information as necessary, considering the rate of change and the importance of historical data. Satellites Data Loading Strategies : Depending on your data volume and frequency of updates, choose between full loads (reloading all data) and incremental loads (loading only new or changed data) to optimize performance. Full Loads vs. Incremental Loads Managing Historical Data and Changes Over Time Implementing a robust strategy for tracking historical changes is crucial. Use Satellites to record time-variant attributes and employ effective dating techniques to keep track of historical data. Following these best practices and considerations will help ensure a smooth implementation of Data Vault modeling in your organization, leading to a flexible, scalable, and future-proof data warehouse. Example of Data Vault Modeling To better understand how Data Vault modeling works in practice, let's walk through a simplified example involving customer orders. This example will help illustrate the structure and relationships between Hubs, Links, and Satellites in a Data Vault model. Scenario Overview Consider a retail company that processes customer orders. We want to model data involving customers, products, and orders. The Data Vault model will include three Hubs (Customer, Product, Order), one Link (Order Details), and Satellites to store descriptive and historical information. Hubs Hubs represent the core business concepts. In this scenario, we have: Hub Customer (Hub_Customer) Column Name Description HUB_ID Unique identifier CUSTOMER_ID Business key (ID) LOAD_DATE Date of record entry RECORD_SOURCE Source of data Hub Product (Hub_Product) Column Name Description HUB_ID Unique identifier PRODUCT_ID Business key (ID) LOAD_DATE Date of record entry RECORD_SOURCE Source of data Hub Order (Hub_Order) Column Name Description HUB_ID Unique identifier ORDER_ID Business key (ID) LOAD_DATE Date of record entry RECORD_SOURCE Source of data Link Links model the relationships between Hubs. In this case, we have: Link Order Details (Link_OrderDetails) Column Name Description LINK_ID Unique identifier ORDER_HUB_ID Foreign key to Hub_Order PRODUCT_HUB_ID Foreign key to Hub_Product CUSTOMER_HUB_ID Foreign key to Hub_Customer LOAD_DATE Date of record entry RECORD_SOURCE Source of data Satellites Satellites provide descriptive details and historical information about Hubs and Links. Satellite for Customer (Sat_CustomerDetails) Column Name Description HUB_ID Foreign key to Hub_Customer NAME Customer's name EMAIL Customer's email LOAD_DATE Date of record entry RECORD_SOURCE Source of data Satellite for Product (Sat_ProductDetails) Column Name Description HUB_ID Foreign key to Hub_Product NAME Product name DESCRIPTION Product description PRICE Product price LOAD_DATE Date of record entry RECORD_SOURCE Source of data Satellite for Order Details (Sat_OrderInfo) Column Name Description LINK_ID Foreign key to Link_OrderDetails QUANTITY Quantity ordered ORDER_DATE Date of the order DELIVERY_DATE Expected delivery date LOAD_DATE Date of record entry RECORD_SOURCE Source of data Continuing from our example of Data Vault modeling with customer orders, let's delve into how these models are used in practice and the benefits they offer. Using the Data Vault Model in Practice Querying Customer Orders Suppose you want to generate a report showing all orders placed by a specific customer along with product details. Using the Data Vault model, you would: : Use to find the corresponding to the customer of interest. Identify the Customer Hub_Customer HUB_ID : With the customer's , query to retrieve all s associated with the customer. Find Related Orders HUB_ID Link_OrderDetails ORDER_HUB_ID : For each , look up to gather details about each order, such as quantity and order date. Fetch Order Details ORDER_HUB_ID Sat_OrderInfo : Use from to join with for product names, descriptions, and prices. Retrieve Product Information PRODUCT_HUB_ID Link_OrderDetails Sat_ProductDetails Analyzing Product Sales Over Time To analyze how product sales have varied over time, you can: : Query and to collect quantities and order dates for all products. Aggregate Order Data Link_OrderDetails Sat_OrderInfo : Use to join with for product names. Join with Product Information PRODUCT_HUB_ID Sat_ProductDetails : Group the aggregated data by product name and time period (e.g., month, year) to analyze trends and patterns in product sales. Time-based Analysis Benefits of Using Data Vault Models in Practice Flexibility in Querying Data Vault's separation of business keys (Hubs), relationships (Links), and descriptive information (Satellites) offers unparalleled flexibility in querying data. Users can easily navigate between different aspects of the data, combining them in various ways to answer complex business questions. Scalability As new data sources are integrated, new Hubs, Links, or Satellites can be added without disrupting existing structures. This modular growth capability ensures that the data warehouse can scale alongside the business. Auditability and Data Governance Each component in a Data Vault model includes and , providing a clear audit trail of when data was added and its source. This feature supports rigorous data governance practices and compliance requirements. LOAD_DATE RECORD_SOURCE Handling Historical Data The Data Vault model excels at managing historical data through Satellites. Changes over time are captured in detail, allowing businesses to track trends, perform historical analyses, and make informed predictions. Enhanced Data Quality By clearly separating different aspects of data and maintaining detailed metadata, Data Vault models help in identifying and rectifying data quality issues. This structure promotes high data integrity and reliability for reporting and analytics. Conclusion The practical use of Data Vault models, as illustrated through querying customer orders and analyzing product sales, demonstrates the methodology's strengths in flexibility, scalability, and data governance. By adopting Data Vault modeling, businesses can build a robust foundation for their data warehousing and analytics needs, capable of adapting to change and supporting complex data landscapes. Also published . here

Intro to Data Vault Modeling: Agility, Scalability, and Practical Applications Explained

One of the biggest and most common trends currently seen in organizations is the shift from the traditional means of production to a much more reliable means known as . Agile According to , over the past 25 to 30 years, agile innovation methods have led to a revolutionalized information and technology sector, resulting in an increased success rate in software development, quality, and speed to market, which in turn boosts motivation and productivity of team members. Havard Business School Principles of Agility breathe life into software development, fostering flexibility, teamwork, and continuous progress. In DevOps, Agile smashes traditional barriers, encouraging collaboration and embracing change. Short development cycles, constant feedback, and adaptive planning characterize Agile practices. It's a far cry from developers vanishing behind closed doors for months; instead, they engage with operations throughout. In DevOps, Scrum and Kanban shine. Scrum uses structured sprints and daily meetings, ensuring transparency and organized tasks. Meanwhile, Kanban visualizes tasks, identifies bottlenecks, and enhances efficiency, all while maintaining a steady development flow. Agile principles Harnessing the Power of  Control in DevOps In the realm of DevOps, control is the guiding force ensuring harmony, stability, and safety in software development. It acts as a guardian, ensuring that software operates seamlessly while encouraging innovation. Control is the linchpin for stability, preventing disruptions and crucial failures. It takes shape through practical methods: : These automated pathways form the foundation of structured development, guaranteeing consistency by automating building, testing, and deployment. They accelerate releases without compromising rigorous testing. CI/CD Pipelines Automated Testing: Bringing predictability to the forefront, automated testing systematically validates code changes. This reduces human errors and catches bugs before they impact the live environment. Monitoring: Like a vigilant guardian, monitoring tools oversee production systems, ensuring their proper behavior. Swift corrective actions can be taken upon detecting any deviations, maintaining the integrity of the software landscape. Balancing Agility and Control Finding the Right Balance between Agility and Control Organizations aiming for excellence often struggle to balance agility and control effectively. Agility thrives on quick updates, adaptability, and innovation, while control focuses on stability, security, and standards. Striking the right balance is crucial. Too much agility can lead to chaos and compromised security, while too much control stifles innovation and slows progress. Successful organizations find harmony between the two, innovating swiftly while ensuring stability, security, and compliance. Ways Harmonizing Agility and Control Can Be Achieved in DevOps By adopting a set of best practices, organizations can seamlessly integrate agility and control, reaping both benefits without compromising on quality or security. Some of the ways balance can be achieved are discussed below. Foster a Collaborative Culture Encouraging open communication between development and operation teams fosters a shared understanding of goals and challenges. enables teams to collectively make informed decisions that balance agility and control. Regular meetings, joint planning sessions, and knowledge sharing enhance teamwork and build a foundation of trust. Cross-functional collaboration Aligning DevOps Principles with Organizational Goals DevOps practices should be aligned with the overarching objectives of the organization. When both agility and control are in sync with business goals, decisions become more purposeful. For instance, if security is a top priority, robust security measures, even if it slightly affects the speed of development. This alignment ensures that agility and control work together to drive business success. DevOps practices should emphasize Choosing the Right Tools for a Smooth Blend Picking the right tools is key to balancing agility and control. Opt for DevOps-friendly tools that automate, collaborate, and watch over the process, making things efficient yet controlled. Tools that show your development journey, automate tests and handle deployment orchestration streamline tasks so quality remains high while speed increases. Boosting Efficiency and Control with Automation Automation is the backbone of both agility and control; It speeds up work and cuts down on errors by automating repeated jobs. It also keeps things standardized, ensuring control over the journey. Auto-tests, seamless integration, and deployment plans guarantee that changes are triple-checked before they go live. Building Strong Security without Killing Agility Security isn't an afterthought; it's part of DevOps DNA. Start early, building security into each development step. Auto security tests, scans for weak spots, and code checks find and fix issues pronto. When security is woven in, agility doesn't mean exposing your system – it's a win-win. Tools and Technology Required for Effective Harmonization Finding the right balance between agility and control is a key challenge. Luckily, tools and technologies are available to guide organizations through this fine balance. Here are some crucial tools that can lead to significant improvements: Automation Tools Streamlining tasks and reducing errors, these tools handle both development and operations. This frees up teams for innovation rather than repetitive work. From automating deployment to managing resources, tools like Jenkins, Ansible, and Puppet empower teams to deliver dependable software quickly and precisely. Version Control Systems Central to collaboration, these systems ensure smooth teamwork amid ongoing changes. By tracking code changes and facilitating feature branches, systems like Git enhance code quality and teamwork. Developers can work concurrently on various project aspects without conflicts, promoting agility and control. Infrastructure as Code (IaC) Infrastructure as Code (IaC) involves managing and provisioning infrastructure using code instead of relying on manual processes. IaC empowers teams to shape and oversee infrastructure using code, reducing differences among development, testing, and production setups. Tools like Terraform, AWS, and Cloud Formation automate resource setup, curbing configuration differences and enabling reliable, repeatable changes. This balances stability and modern flexibility in development. Conclusion In a harmonious organization, both agility and control thrive through strong collaboration between agile teams and operations. This collaboration unlocks boundless potential. Yet, for organizations transitioning from outdated methods, achieving this balance can be exceptionally tough. It's a complete shift from the old ways. This article presents a clear guide to achieving this harmony, ensuring seamless operations and peak productivity.

Striking the Balance: Harmonizing Agility and Control in DevOps

How do you deal with uncertainty? How do you handle unexpected events? How do you react to change? Do you panic and avoid dealing with the situation or do you look at the situation with curiosity, explore alternative ways to move forward, and pivot, if need be? The ability to think on your feet and solve problems by quickly moving between different ideas is the key to creativity at work. Mental agility makes you confidently step up and solve difficult problems when everyone else is taken aback by an unpredictable situation or an unforeseen circumstance and wondering how to deal with it. Mental agility is necessary to innovate, embrace change, and thrive in the face of uncertainty. Building mental agility requires radically transforming your thinking for both speed and accuracy. It involves the ability for smarter reasoning by asking meaningful questions, drawing insights, and analyzing situations with better mental clarity. You can’t play out every scenario in your mind and plan for it. You can’t predict everything that might come your way. However, you can learn to think in a flexible way—accept change, find the next best course of action, and quickly move forward instead of being stuck in one place. It is not the strongest or the most intelligent who will survive but those who can best manage change ― Steve Magness Many people mistake mental agility with toxic positivity—they deny their emotions and keep pushing forward no matter what comes their way. Mental agility is not toxic positivity. It is optimism blended with reality. Your default wiring to fight and resist change makes an unexpected situation feel like a threat. Unless you consciously fight the urge to give in to your emotions, feelings of helplessness that such situations tend to evoke will make you mentally rigid and shut down thinking. If you want to be present to the needs of a changing workplace and thrive in uncertainty, follow these five key practices to achieve gains in performance, develop greater focus, and enhance thinking. 5 Practices to Build Mental Agility 1. Purposefully step outside your comfort zone When you do things you have always done or the ones that make you comfortable, you give limited opportunities to your brain to grow. Your brain learns to associate certainty with safety and anything uncertain as a threat. Sitting within the bounds of your comfort zone makes mental agility impossible—how can you think quickly in an unpredictable situation when your mind is engulfed with fear, anxiety, and worry? Pushing yourself to do new things or the ones that make you uncomfortable tunes your brain to accept discomfort as a part of life—you stop seeing change as a threat, become more open to diverse ideas, and learn to navigate complexity instead of avoiding it. When your brain is not overwhelmed by strong emotions, it can think clearly. You can move quickly between different ideas and respond to events in a flexible way. To step outside your comfort zone: Challenge yourself to solve hard problems. Seek opportunities that require you to solve problems that you have never solved before. Learn something new. It doesn’t even have to be work-related. How about the piano classes, tennis lessons, mastering French, or taking cooking classes that you keep putting off? Set yourself a difficult goal and work on it consistently. For example: push yourself to exercise for 30 days without a break. Real change is difficult at the beginning, but gorgeous at the end. Change begins the moment you get the courage and step outside your comfort zone; change begins at the end of your comfort zone. ― Roy T. Bennett Your brain loves a challenge, and challenging it frequently will keep it mentally agile. 2. Practice Finding Multiple Answers When finding a solution to a problem, most of us settle for the first best idea that comes to mind. We hardly take the initiative to explore alternatives or consider other viewpoints. Thinking a certain way or being rigid about your choices limits your brain’s potential to think creatively—if your brain has been trained to think inside a box for too long, it can’t imagine the boundless possibilities. Mental agility requires building curiosity to seek multiple solutions to a problem, weighing their pros and cons, identifying the best one in the moment, taking action, and moving on. Doing this for problems you face on a regular basis makes this process automatic—it becomes a habit. Your brain learns to quickly absorb new information, draw connections between ideas and find the best course of action to move forward. To get into the habit of finding multiple answers to a problem: Challenge yourself to always think of more than one solution to a problem. Encourage, invite, and show curiosity to hear others’ points of view. Don’t reject an approach just because it seems counterintuitive or different from your approach. Use an inversion mental model to think the opposite of what you seek. It will help you uncover issues that you didn’t consider before. Shift from scarcity to abundance mindset. An abundance mindset opens your mind to new opportunities by viewing the world with infinite possibilities and unlimited options. When you’re not operating within the confines and constraints of your own limitations, you no longer see the world as a limited pie but view it as an endless ocean where there’s enough for everyone. To build mental agility, remind yourself that there’s always more than one solution to a problem. Don’t settle till you have exercised enough muscles in your brain and thought hard about the problem. 3. Develop Creative Thinking Skills Remember all that curiosity you had as a child that annoyed your teachers and parents when you asked too many questions: “Why do I have to go to school?” “Why do I have to sleep early when you can be awake till late?” “Why can’t I play video games?” “Why do I have to finish my homework?” Curiosity and the ability to express that curiosity constructively is a great skill to have, even when you’re an adult. It allows you to look at things differently and come up with new solutions to problems. Without curiosity, you accept the status quo and assume things only work a certain way. You can’t be mentally agile if you keep your curiosity hidden and fail to unleash it. To build creative thinking skills, wake up your curious mind and put it to use: Challenge and question assumptions. Expand your circle of competence by gaining knowledge from outside your current scope of work. It will enable you to combine various types of information in your head in new and novel ways. For example: meet with other functions within your organization to understand how they operate, what their challenges are, and how they make decisions. Create mental space for new ideas to kick in. Incorporate thinking time into your calendar. Creativity isn't a switch that's flicked on or off; it's a way of seeing, engaging and responding to the world around you ― Rod Judkins Curiosity didn’t kill the cat. If you want to be mentally agile, get curious. 4. Spend Mental Energy Consciously Have you recently taken notice of how you spend your mental energy? Do you spend more time thinking about things you can’t control or what you can do to make things better? Focusing on events, circumstances, and conditions beyond your control makes you feel powerless and helpless—your mind gets busy obsessing about the problem and fails to consider the possibilities. When encountered with an unpredictable situation, directing your energy towards things you can’t control is easy—you need no effort as your brain is already good at it. By default, it chooses the path of least effort, which is to blame it on someone or something else and absolve itself from its responsibility. Shifting your default response to act constructively requires a conscious effort. Instead of worrying about what you cannot control, shift your energy to what you can create. ― Roy T. Bennett To spend mental energy productively: Start with taking responsibility. Identify what’s within your control—even in the most difficult circumstance, there’s always something you can do. Take small steps in its direction. Learn from your actions, change, adapt, and repeat. When you’re conscious about how you spend your mental energy and tune it to align with more positive habits, you stop wasting time ruminating and shift to action. You build mental agility. 5. Build a Growth Mindset One belief that can often get in the way of building mental agility is the mindset, “This is who I am.” “This is how I think.” “I can’t get any better.” When you believe that your personality is fixed and no amount of effort can change it, you already accept defeat. Carol Dweck, a Stanford psychologist, calls it a fixed mindset. As I explain in my book a person with a fixed mindset believes that people are born with special talents and every person has different abilities and intelligence that cannot get better with time, effort, and determination. A growth mindset person, on the other hand, understands that certain people have special talents and that intelligence varies from person to person, but it’s also something that can be developed and increased with effort and hard work. \ Upgrade Your Mindset, Your personality is nothing but a result of this mindset. A fixed mindset prevents you from fulfilling your potential by making you believe that you can’t change over time. A growth mindset, on the other hand, keeps you learning and growing by giving you the power to shape yourself into the person you wish to become. Research from Carol Dweck and multiple other psychologists has shown that even our deepest psychological traits can change over time with deliberate interventions. With effort and perseverance, you’re no longer limited by who you’re; you can grow into the person you wish to become. Mental agility is all about being able to adapt to uncertainty, which first and foremost requires the belief that you’re capable of learning, growing, and developing. To build a growth mindset: Shift to empowering language that encourages you to act. Set learning goals as opposed to performance goals. Performance goals focus on self-validation: performing well to prove that you’re good at what you do. Show that you’re smart, talented, and capable or to outperform other people. Learning goals focus on self-improvement: progress, growth, and gaining mastery. Becoming the best, most capable person you can be rather than proving that you already are. Capitalize on your failures. Treat them as a means to get better instead of considering them as a limitation of your abilities. Choose Goldilocks tasks—activities that are neither too easy nor too difficult, just a little over your current abilities. They provide a perfect opportunity to step outside your comfort zone without causing anxiety. For more strategies on growth mindset, check out my book Upgrade Your Mindset, which provides practical strategies to build a growth mindset for life. Summary When dealing with an unpredictable situation, an unexpected change or uncertainty, what do you need to identify the best way to move forward? Mental agility. Mental agility enables you to move quickly between different ideas and respond to events in a flexible way. To build mental agility, purposefully step outside your comfort zone. It will align your brain to accept discomfort as a part of life instead of avoiding it. Instead of limiting your thinking with the first solution that comes to mind, challenge your brain to seek multiple answers. Expand your creative thinking skills by challenging assumptions, gaining knowledge outside your domain, and creating mental space for new ideas to kick in. Consciously shift your mental energy to things within your control. It stops rumination and leads to action. Finally, none of this is possible if you don’t believe in your ability to change. Believe in the power of a growth mindset and practice strategies to get there. Previously published . here

Learn How to Cultivate Mental Agility and Thrive in Uncertainty With These 5 Tips

The BFF pattern optimizes backends for specific client interfaces. Using GraphQL for it fixes some of its drawbacks, but can we do better? Let’s find out with WunderGraph. The Backends-for-frontends (BFF) pattern, , and first used at SoundCloud, is an effective solution when you have multiple client platforms  -- web browsers, mobile apps, gaming consoles, IoT devices  --  each with unique requirements for data, caching, auth, and more. first described by Sam Newman The BFF is a custom interface (maintained by the frontend team) that is purpose-built for the needs of each client platform, serving as the only “backend” as far as the client is concerned. It sits between the client and underlying data sources/microservices, keeping them decoupled, and giving each client team full control over how they consume, consolidate, and massage data from downstream API dependencies  -- and handle their errors. But the BFF pattern, like anything else in engineering, has its tradeoffs and drawbacks. As it turns out, though, GraphQL is great at shoring up these flaws, and a GraphQL-driven BFF layer works well. However, GraphQL brings with it its own set of problems! Starting to sound like a broken record, here. How do we break the cycle? Let’s talk about it, with a look at a free and open-source technology  -- -- that can help us. WunderGraph Backends-for-Frontends ❤ GraphQL BFFs afford you the luxury of maintaining and securing only a single entry point as you do in simple monoliths, while still working with microservices internally. This gives the best of both worlds, but that kind of agility comes at a price. Let’s take a look at some of these issues, and how GraphQL nullifies them: BFFs work great as an aggregation layer, but you still need multiple requests to get the data you want, and this scales linearly with the number of Microservices/API/Data dependencies you have. Multiple network hops add latency. GraphQL works great in minimizing multiple round trips because it natively enables clients to request multiple resources and relationships in a single query. Overfetching and underfetching might still be problems because of a combination of how RESTful architectures inherently operate, and the fact that you may be relying on legacy or third-party APIs whose architectures you do not own. Image Source: https://www.howtographql.com/basics/1-graphql-is-the-better-rest/ In RESTful APIs, it’s the server that determines the structure and granularity of the data returned, and this can lead to more data than the client actually needs. GraphQL shifts that responsibility to the client, and they can specify exactly the data they need in a single query. Image Source: https://www.howtographql.com/basics/1-graphql-is-the-better-rest/ Whether or not your downstream microservices/APIs are using GraphQL themselves doesn’t matter for the client/BFF interaction. You won’t need to rewrite anything to reap the benefits. 3. Documentation is going to be an ongoing chore. A BFF is purpose-built for a specific client, and so each BFF will need detailed accompanying API documentation (with payload/response specifications) to be created and kept up-to-date. If using REST, the endpoint names and HTTP verbs will only grow with the app and its complexity, making them difficult to document, and thus, difficult to onboard new hires for. GraphQL is declarative and self-documenting by nature. There’s a single endpoint, and all available data, relationships, and APIs can be explored and consumed by client teams (via the or just ) without constantly going back and forth with backend teams. GraphiQL interface Introspection …but GraphQL Isn’t a Silver Bullet. Adopting GraphQL shores up some BFF issues, no doubt, but GraphQL adoption isn’t something to be taken lightly. It’s not just another UI library to add to your tech stack, and much like anything in engineering, it has growing pains, trade-offs, and complexities associated with it. GraphQL has a steep learning curve. Understanding its concepts  -- schemas, types, queries, mutations, and subscriptions  --  requires a substantial time investment, and writing good resolver functions requires a good understanding of the underlying data models and how to retrieve and manipulate data from different sources (databases, APIs, or services). Server-side data coalescing is a neat idea, but this makes your GraphQL implementation only as good as your resolvers and involves a ton of boilerplate that can make the implementation more verbose and error-prone. GraphQL has no built-in caching. The simplicity that GraphQL’s single endpoint format brings to the table can also become a bottleneck in certain circumstances  --  REST APIs’ multiple endpoints allow them to use HTTP caching to avoid reloading resources, but with GraphQL, you’re only using one universal endpoint for everything and will have to rely on external libraries (like Apollo) for caching. GraphQL clients can be pretty heavy in terms of bundle size that you have to ship. This might not matter in some use cases (if you’re only ever building an internal app, for example) but it can and does impact the load time and performance of your application, especially in scenarios where low bandwidth or slow network connections are a concern. Performance can be a concern if you’re not careful. The nature of GraphQL makes it non-trivial to harden your system against drive-by downloads of your entire database. It’s far more difficult to enforce rate limits in GraphQL than it is in REST, and a single query that is too complex, too nested, fetching too much data, could bring your entire backend to a crawl. Even accidentally! To mitigate this, you’ll need an algorithm to calculate a query’s cost before fetching any data at all  -- and that’s additional dev work. (For reference, this is how GitHub’s GraphQL API does it ) GraphQL doesn’t have a built-in way to ensure security. REST’s vast popularity and authentication methods make it a better option for security reasons than GraphQL. While REST has built-in HTTP authentication methods, with GraphQL, the user must figure out their own security methods in the business layer, whether authentication or authorization. Again, libraries can help, but that’s just adding more complexity to your builds. Introducing WunderGraph. WunderGraph is a free and open-source (Apache 2.0 license) framework for building Backends-for-Frontends. GitHub - wundergraph/wundergraph: WunderGraph is a Backend for Frontend Framework to optimize… _WunderGraph is a Backend for Frontend Framework to optimize frontend, fullstack, and backend developer workflows through…_github.com Using WunderGraph, you define a number of heterogeneous data dependencies  -- internal microservices, databases, as well as third-party APIs  --  as config-as-code, and it will introspect each, aggregating and abstracting them into a virtual graph. namespaced Here’s what a system you might build with WunderGraph would look like. Let’s dive right in. First of all, you’ll have to name your APIs and services as data dependencies, much like you would add project dependencies to a file. package.json ./.wundergraph/wundergraph.config.ts // Data dependency #1 - Internal database  
const db = introspect.postgresql({  
 apiNamespace: "db",  
 databaseURL: "postgresql://myusername:mypassword@localhost:5432/postgres",  
});  
  
// Data dependency #2 - Third party API  
const countries = introspect.graphql({  
 apiNamespace: "countries",  
 url: "https://countries.trevorblades.com/",  
});  
  
// Add both to dependency array  
configureWunderGraphApplication({  
 apis: [db, countries],  
}); You can then write GraphQL operations (queries, mutations, subscriptions) for getting the data you want from that consolidated virtual graph. ./.wundergraph/operations/CountryByID.graphql query CountryByID($ID: ID!){  
 country: countries_country(code: $ID) {  
 name  
 capital  
 }  
} ./.wundergraph/operations/Users.graphql query Users($limit: Int!){  
 users: db_findManyusers(take: $limit) {  
 username  
 email  
 }  
} …and WunderGraph will automatically generate endpoints that you can cURL to execute these queries and return the data you want, over . JSON-RPC > $ curl http://localhost:9991/operations/CountryByID?ID=NZ  
> {"data":{"country":{"name":"New Zealand","capital":"Wellington"}}} > $ curl http://localhost:9991/operations/Users?limit=5  
>{"data":{"users":[{"username":"user1","email":"user1@example.com"},{"username":"user2","email":"user2@example.com"},{"username":"user3","email":"user3@example.com"},{"username":"user4","email":"user4@example.com"},{"username":"user5","email":"user5@example.com"}]}} If you’re using data fetching libraries like or , or a React-based framework (NextJS, Remix, etc.) you get even more out of WunderGraph, as it auto generates a lean, performant, custom TypeScript client for you every time you define an operation, complete with fully type-safe hooks for querying and mutating data that you can use on the frontend, wherever you need it. SWR Tanstack Query ./pages/index.tsx // Import data fetching hook from WunderGraph-generated client  
import { useQuery } from "../components/generated/nextjs";  
  
const Home: NextPage = () => {  
 // Data dependency #1 - Internal Database  
 const { data: usersData } = useQuery({  
	 operationName: "Users",  
	 input: {  
	 limit: 5,  
	 },  
 });  
  
 // Data dependency #2 - Third-party API  
 const { data: countryData } = useQuery({  
	 operationName: "CountryByID",  
	 input: {  
	 ID: "NZ",  
	 },  
 });  
 
//...  
} Now, you can use and however you want. Render it out into cards, lists, pass it on to props and go wild. usersData countryData But How Does WunderGraph Address GraphQL’s Issues? Easily  -- . WunderGraph takes GraphQL completely out of the runtime Yes, you do write GraphQL operations to collect and aggregate the data you want from multiple data sources… but that’s it. You won’t have to write resolvers or worry about making them efficient at data fetching. Your involvement with GraphQL goes only as far as writing queries, and only during development. No public GraphQL endpoint is ever exposed, nor is any heavy GraphQL client shipped on the client. Don’t worry if this doesn’t click for you immediately. I promise it’ll make sense soon. Let’s look at this process in detail. GraphQL only during local development? Whenever you write a GraphQL query, mutation, or subscription and hit save, the Query is NOT saved as is. (together with input variables, if needed, of course). Instead, it is hashed and then saved on the server, only able to be accessed by the client by calling for the hashed name This is what is called a . Your WunderGraph BFF server has a list of known hashes  --  as many Persisted Queries as you define  -- and will respond to client requests for valid hashes, serving them as JSON over RPC, with a unique endpoint for each and every operation. Persisted Query only (And WunderGraph generates typesafe hooks for React frameworks and data-fetching libraries that do the same thing) What does this mean? Two things. You address the security concerns associated with having your GraphQL API reverse-engineered from browser network requests  -- because there is no GraphQL API. If each operation has a unique endpoint, you can finally implement traditional REST-style caching! But that’s not all. The WunderGraph BFF server generates an for each response  --  this is a unique identifier for the content of said response  -- and if the client makes a subsequent request to the same endpoint, and its ETag matches the current ETag on the server, it means the content hasn’t changed. Entity Tag (ETag) Then the server can just respond with an HTTP 304 “Not Modified”, meaning the client’s cached version is still valid. This makes clientside stale-while-revalidate strategies blazingly fast. You can write all the arbitrary queries you want using GraphQL during development time, but they’re hashed and persisted at compile time and only show up at runtime as JSON RPC. In one fell swoop, you have solved the most common issues with GraphQL, creating a best-of-both-worlds scenario. All the wins of GraphQL, with none of the drawbacks. But what about the performance concerns of too-complex queries? WunderGraph uses the internally, crafting optimized queries under the hood for each Operation you write, no matter your data source. Prisma ORM You won’t have to worry about hand-rolling your own SQL or GraphQL queries and sanitizing them to make sure a single naïve query doesn’t kill your downstream services, or lead to DOS attacks. Where to Go From Here? You’ve seen how WunderGraph makes building BFFs so much easier, with fantastic devex to boot. Using Prisma under the hood, it creates persisted queries and serves them over JSON-RPC, reining GraphQL back to only being used in local development. But you can do far, far more than that with WunderGraph --  it’s a true BFF framework: It supports fully typesafe development from front to back. Every Operation you write will generate a typesafe client/hook to access it from the frontend…and ! With WunderGraph, you have shared types between the backend and the frontend  --  giving you full IDE autocomplete and inference both while writing GraphQL queries, and when developing the client. you can even generate typesafe Mocks this way It can turn any query you write into a without even needing WebSockets  --  making WunderGraph ideal for building BFFs that get deployed together with the frontend to serverless platforms, which use WebSockets. Live Query can’t It can also integrate , OIDC/non-OIDC compatible auth providers, and more. S3-compatible storage for file uploads If you don’t want to write GraphQL at all, you can write custom --  async resolvers, essentially  -- to define Operations during development. You can perform JSON Schema validation within them using Zod and aggregate, process, and massage the data any way you want, and then return it to the client. TypeScript Operations are accessed exactly like GraphQL Operations are (JSON RPC endpoints, or clientside hooks), and they run entirely on the server and are never exposed clientside. Plus, you can even import and use existing GraphQL operations within them. TypeScript functions You can bake cross-cutting concerns that are ideally present on the BFF layer  -- most importantly, auth  --  into WunderGraph’s BFF layer via / /your own auth solution, and every Operation can be authentication aware. Clerk Auth.js With powerful introspection that turns API, database, and microservices into dependencies, much like a package manager like NPM would, WunderGraph makes BFF development accessible and joyful, without ever compromising on end-to-end type-safety. To know more, check out their docs , and their Discord community . here here Also published here

Unleashing Backend Agility: Empowering Your App With Build-Time GraphQL

If you're not familiar with the episode from Seinfeld, the tl;dr of it is that George's father, Frank, is given the mantra of " to help curb his enthusiasm for angry outbursts. Impressed with its effectiveness, Kramer adopts the mantra for himself. George's childhood rival, Lloyd Braun, later cautions that this mantra doesn't address feelings, but bottles them up, and leads to huge explosive anger. And that's when he delivers the classic line: Serenity Now Serenity now..." Serenity now. Insanity Later. At the surface, it's valuable advice about controlling our emotions, and stress - but deep within this statement is also a very important guideline for delivering Agile software. And while Seinfeld isn't known for technology, it's too perfect that the backdrop of this episode has Frank, George, and Lloyd selling desktop computers. George, of course, struggles to keep up with the zen-like Lloyd, who has reached true Agile enlightenment. In fact, let's add further insult to George's unending injuries. I think Lloyd deserves to have an Agile principle named after him. Arguably, Lloyd is the given that this episode first aired in 1997 - some 4 years before seventeen people would meet at a ski resort in Utah to produce the . grandfather of Agile Agile Manifesto But before looking deeper into Lloyd's principle, let's first revisit the original Agile Manifesto's 12 principles: The highest priority is to satisfy the customer through early and continuous delivery of valuable software Welcome changing requirements, even late in development. Harness change for competitive advantage. Deliver working software frequently. Business people and developers must work together throughout the project. Build projects around motivated individuals with the environment and support they need to get the job done. Prioritize face-to-face conversations. Working software is the primary measure of progress. Promote sustainable development, where a constant pace can be supported indefinitely. Attention to technical excellence and good design to enhance agility. Preference for building on things_._ not Build self-organizing teams. Reflect on how to become more effective, and adjust. Taking a closer look at each of these, they all share the same spirit around problems we face regularly: We know that software systems only increase in complexity with time. Technical debt grows, new features creep in, scope increases, software gets bloated, developers burn out, people get frustrated, and quality decreases. You know it all. And that's where I think the matters. It matters that I believe in , with Lloyd's getting placed at index 0. Lloyd Braun Principle of Agility so much 13 Agile Principles 0. Simplicity now - complexity later; Serenity now, insanity later. There are two lessons to be learned from this one principle. The first lesson comes from the first part of the statement: We know not to over-engineer. We deliver to the baseline. The 80/20, KISS type rules. Deliver the MVP. Prioritize value. Add complexity later, and only if there's demand for it. Serenity now. But the second lesson tucked away in the second part - that is what we don't spend enough time discussing. It's not just an instruction to add complexity later - to put off the insanity. It is a warning that prioritizing simplicity means insanity come. Whether it's in explosive user growth, challenging timelines, taking on technical debt in favor of delivering value early on or maintaining legacy code that didn't scale to meet insane needs. Insanity and complexity are to be expected. Insanity later. will explains the first 6 Agile principles. Those first six offer a framework for delivering with simplicity. Serenity now is dealt with in those bottom 6 principles. Those principles help us manage the chaos: How we converse, how we measure value, how we should push back, how we should organize, and continuously evaluate and be critical of how we are working. Insanity later I hear too much about retros that don't happen or aren't taken seriously . Or I've seen too many teams that are resistant to change, or adapting Or, worst of all, managers who subvert what it means to have a self-organizing/self-managing team or who measure progress by output and not outcome. ("There's no point in bringing this up... it's not like my manager can fix it...") ("Why are we changing the process again? I thought this was working?"). Especially in this new world, we're living in - with remote teams, virtual meetings, and a renewed emphasis on work/life balance - I think it's high time we take a more holistic view of what it means to be Agile - focusing less on serenity now, and more on the insanity later. Also published here.

The Lloyd Braun Principle of Agility

Why Implementing Microservices on AWS Is Indispensable to a Modern Architecture for Business Agility

Agile stepped in when linear requirement-based processes were playing around to impose project management processes and predictable delivery methodologies across complex software domain, driven by time and output. Even with a certain percentage of success, Agile hasn't managed to kill off the embedded waterfall behaviors, prescribed linear plans and over-complex management layers for delivering software en masse. Going forward, Scrum and Extreme Programming (XP) have turned out to be the most trusted and established agile software methodologies for building software in dynamic environments. The agile methodology does not resolve every software issue, but these act as a cue stick for relatively small, self-organizing teams. Introducing: The Scaled Agile Framework (SAFe) Organizations have propelled their efforts to imbibe The Scaled Agile Framework (SAFe) that is meant to address scaling Agile to the enterprise level, as it quickly combines Agile approaches with more enterprise-centric organizational practices. But as enterprises undergo the broader change necessary for digital transformation success, SAFe still faces certain challenges. Scaled Agile Framework (SAFe) is an interactive knowledge base for implementing the at an enterprise scale. It is difficult to understand the scope of SAFe for scaling agile software development to the enterprise. Agile development practices are not limited to Lean Startup, Right Shifting, Beyond Budgeting, Management 3.0, Human-Centered Design, Theory of Constraints, Cynefin, Portfolio Management, etc. Agile model How can Agile Planning turn the tides in favor of organizations? adoption, willingness to throw out a plan, know your minimum viable product, ship it, gather feedback, iterate. It doesn’t have to be perfect. Just good enough for the next development cycle. Agile implies: Agile organizations have an extra ability to manage changing priorities, improved visibility, increased productivity, improved quality and reduced risk. All this leads to customer delight, joy at work, engagement, innovation, creativity and continuous learning etc. Agile organizations face the challenge of managing the local autonomy of squads (bottom-up input) consistently with a bigger picture represented by the tribe’s goals and by cross-tribe interdependencies and the strategic priorities of the organization (top-down view). Governing this tension requires new processes and routines for planning and coordination. Associated frameworks and tools are always capable of: The need for quality time to be invested for a true strategic conversation rather than simply being a numbers game; Resources and funds are available in a flexible way for emerging opportunities. And the ability to cope with more frequent and dynamic changes; Prepping-Up Enterprises for more “Agility” As the discussion unfolds, scrums, grand-masters, stand-ups, and sprints help organizations align, co-create and collaborate and leverage more operations and make their business move, let’s find out how to pursue this complex goal of adopting Agile: Let knowledge move across all layers of the organization. Create a platform to network, communicate, crowd-source the issues. Introduce and inspire self- learning and delegating the knowledge to staff. Knowledge Dispersion: Creating project teams across various business verticals (HR, Marketing, Sales and Promotion, Automation, Testing, Development, etc.), for more successful adoption of Agile based around a specific customer or project. Bringing Various Skills on the Table: Project teams allows all stakeholders to be brought into the process early. This creates a true sense of collaboration resulting in a greater sense of ownership from beginning to delivery with fewer compromises. Making a Team Build Its Efforts: Give them the tools, and make them learn. This process brings the required agility into the business. Empower your staff: Daily stand-ups are important as are an analysis of daily hurdles is important to learn and become proficient. It also helps break your production timeline into iterations, organize face to face meetings and measuring success at the end of the process. Assign Daily Tasks and Set Expectations: Agile as a Breather to Traditional Project Management Approaches A certain level of understanding is required to compile the fact that Agile methodologies alone are not sufficient to address top-down questions of business strategy. As Agile teams work bottom-up adding up the opinions of people to come up with the business strategy must be combined with centralized decision making. Companies need to imagine possibilities and pick the one for which the most compelling argument can be made. Compelling arguments need to be backed up with a qualitative judgment. In Agile Software Development organizations , teams make use of design thinking and other exploratory techniques (including data) to make rapid decisions and change the course every week. Decision making is done by a team of people, offsetting in this way the potential biases of a single person on their judgment. To some extent, an agile team-based organization enables the possibility to leverage qualitative data and judgment – combined today with infinite hard data – for better decisions. Get the minimal requirements and start working. Explore more about Agile model here: https://www.konstantinfo.com/blog/agile-model/

AGINITY

aginity.com

11-50 employees

Since 2020

COMPANY RANKING

AGINITY

EVERGREEN INDEX #963

Representing this company? Click here to claim and customize this page!

Aginity WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

AGINITY

aginity.com

11-50 employees

Since 2020

COMPANY RANKING

AGINITY

EVERGREEN INDEX #963

RELATED COMPANIESRANK

STORIES ABOUT AGINITY

Representing this company? Click here to claim and customize this page!

Aginity WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!