IT Managed Services from real people that get it

At ArcSource, we've built long standing relationships with many Bay Area organizations and firms nationwide. Ask them what they appreciate most about our technical and support services and you'll get a range of answers. It could be our fast response time, personalized service, intuitive problem solving skills or the fact that we're approachable, reliable and respectful of their time and work. We offer expertise in Windows and Macintosh networks and specialize in cross platform environments. We have extensive experience in all Microsoft server frameworks. We excel at facilitating productivity for remote workers and mobile device users. In addition, ArcSource is a Google Cloud Partner for Google Workspace (formerly G Suite), an intelligent suite of apps from Google Cloud, including email, calendaring, and collaboration tools. Our focus is on making you more productive using these best-­in-­class tools. The focus of our Managed Care Services is to keep your systems operational and available so that you can focus your efforts on the demands of growing your organization and supporting your clientele. With our Managed Care Services, we provide you with our highest level of support inside a flat-fee monthly agreement. We optionally offer project hours for improvements to the IT infrastructure.

ArcSource Consulting

THE ELECTRIC ARCS

Compromised devices/machines are a new threat to organizations, especially small businesses. Challenges such as Fast-paced digital transformation, Cloud adoption, and the Internet of Things add fire to the already burning pyre of managing the massive volume of machine identities. But this is not all. While the security teams at larger enterprises are fighting demons like ransomware, malware, phishing, and other cybersecurity attacks, small businesses face the significant brunt of poorly managed machine identities. But… 🙋What are machine identities, and how are they different from human identities? 🙋Why is managing machine identities crucial for small businesses? 🙋What are the challenges faced by Small Businesses while managing machine identities? These are a few of the questions I will try to answer through my blog on Managing Machine Identities in SMBs. Sit tight, and keep reading. Machine Identity: The Meaning and The Definition Machine identity is the unique digital identity assigned to applications, services, and devices within a networked environment. These identities help machines to ✅ Connect and communicate with each other in a secure environment. ✅ Authenticate themselves to other systems in the network. Each machine identity comprises cryptographic keys, certificates, or the like of credentials that build trust and enable secure communications in digital space. There are two entities in a networked environment: humans and machines. While much investment is made to protect and manage human identities, scant efforts are made to protect and secure interconnected machines. Result: The imbalance is a significant concern. Reason: The number of machines has steadily increased, outpacing the growth of their human counterparts. This implies that there are more machines connected with each other with scant security efforts in place than humans connected to machines. To protect these machines from being compromised, they need identities that can keep their connections and communications safe. Importance of Machine Identity Management Machine Identity Management plays a crucial role in ensuring your business's security, integrity, and efficiency. With proper identities in place, you can authenticate and authorize machine-to-machine communication. When you are regularly managing the identities of your machines, you can prevent 📌 Unauthorized access 📌 Data breaches 📌 Other security threats, such as malware, ransomware attacks But that’s not all. With machine identity management, you can facilitate the smooth integration of various services and systems. Result: Optimized overall operational performance. But how is this relevant to your small business? Relevance for Small Businesses With limited resources, small businesses face unique challenges in maintaining robust security measures. Despite the size of your business, you still rely on interconnected networks, systems, and applications to conduct your everyday operations efficiently. So, how will managing machine identity help your small business? Managing your machine identities can help you in numerous ways; here are a few prominent ones. Machine Identity Management helps small businesses to 📍Secure their digital assets 📍Safeguard sensitive data 📍Mitigate risks associated with cyber threats With effective machine identity practices, small business owners can ✅ Improve their overall security posture ✅ Support the growth of their business ✅ Scale their small business But what exactly is machine identity? Understanding Machine Identity Unique cryptographic identifiers used to authenticate and secure communication between machines are called machine identities. These identities are allotted to machines in a networked environment, such as servers, devices, and applications. Managing machine identities is crucial for small business owners to protect against cyber threats and maintain their operational integrity in contemporary IT infrastructures. What Constitutes Machine Identity? Machine identity encompasses various components that establish the digital identity of devices, applications, and services within an IT ecosystem. These include 📍Digital Certificates 📍API Keys 📍SSH (Secure Shell) Keys 📍Device Identifiers Why Is Machine Identity Important For Your Small Business Machine identities are crucial for a secure business environment and serve important functions in the contemporary IT infrastructure. 📌 Authentication: Machine identities help business owners verify the identity of devices, applications, and services, ensuring only trusted entities can access resources and systems. 📌Encryption: Machine identities, especially SSL/TLS certificates, enable encryption of data transmitted over networks, safeguarding it from unauthorized interception and ensuring confidentiality. 📌 Access Control: They have a crucial role in access control by regulating permissions based on machines' identities and roles. This helps prevent unauthorized access and actions within IT environments. 📌Understanding and effectively managing machine identities is essential for maintaining IT operations' security, integrity, and efficiency, especially for small businesses with limited resources and specialized needs. Machine Identity Management Challenges for Small Businesses Unmanaged or improperly machine identities can pose serious challenges to small businesses. Some of the challenges your business can face while managing machine identities. Lack of Awareness and Resources Small businesses often face challenges due to limited awareness of available resources and best practices. This can encompass various aspects: 📍 Technology: Small businesses might not be aware of the latest technologies and tools that can help them streamline their operations and improve security. 📍 Financial: Limited budgets can restrict access to necessary resources such as advanced cybersecurity solutions or professional consulting services. 📍 Training and Education: Owners and employees might lack awareness of cybersecurity risks and protocols, leading to vulnerabilities like phishing attacks or weak password practices. Risks Associated with Unmanaged Machine Identities Machine identities refer to digital certificates and cryptographic keys used to authenticate machines in networks. The risks associated with unmanaged machine identities include: 📍Misconfigured or compromised machine identities can lead to unauthorized access to sensitive data or systems. 📍Improperly managed machine identities can result in compliance failures with industry regulations and standards. 📍Unmanaged machine identities can lead to operational disruptions and downtime. Compliance and Regulatory Concerns While compliance with regulations and standards is crucial for your small business, machine identity management can make it challenging due 📍Various regulatory requirements such as GDPR, HIPAA, and PCI DSS can be complex to understand and adhere to, especially for you as a small business owner. 📍Shortage of dedicated resources to meet compliance requirements is another challenge for small business owners. Hiring a dedicated resource can put a strain on your small business budgets. 📍Non-compliance can result in significant penalties or legal consequences. This is quite a detrimental situation for small businesses with limited financial resilience. Small businesses can address these challenges with strategic resource allocation and leveraging technology solutions tailored to their needs. Compromised devices/machines are a new threat to organizations, especially small businesses. Challenges such as Fast-paced digital transformation, Cloud adoption, and the Internet of Things add fire to the already burning pyre of managing the massive volume of machine identities. But this is not all. While the security teams at larger enterprises are fighting demons like ransomware, malware, phishing, and other cybersecurity attacks, small businesses face the significant brunt of poorly managed machine identities. But… 🙋What are machine identities, and how are they different from human identities? 🙋Why is managing machine identities crucial for small businesses? 🙋What are the challenges faced by Small Businesses while managing machine identities? These are a few of the questions I will try to answer through my blog on Managing Machine Identities in SMBs. Sit tight, and keep reading. Machine Identity: The Meaning and The Definition Machine identity is the unique digital identity assigned to applications, services, and devices within a networked environment. These identities help machines to ✅ Connect and communicate with each other in a secure environment. ✅ Authenticate themselves to other systems in the network. Each machine identity comprises cryptographic keys, certificates, or the like of credentials that build trust and enable secure communications in digital space. There are two entities in a networked environment: humans and machines. While much investment is made to protect and manage human identities, scant efforts are made to protect and secure interconnected machines. Result: The imbalance is a significant concern. Result: Reason: The number of machines has steadily increased, outpacing the growth of their human counterparts. Reason: This implies that there are more machines connected with each other with scant security efforts in place than humans connected to machines. To protect these machines from being compromised, they need identities that can keep their connections and communications safe. Importance of Machine Identity Management Machine Identity Management plays a crucial role in ensuring your business's security, integrity, and efficiency. With proper identities in place, you can authenticate and authorize machine-to-machine communication. When you are regularly managing the identities of your machines, you can prevent 📌 Unauthorized access 📌 Data breaches 📌 Other security threats, such as malware, ransomware attacks But that’s not all. With machine identity management, you can facilitate the smooth integration of various services and systems. Result: Optimized overall operational performance. But how is this relevant to your small business? Relevance for Small Businesses With limited resources, small businesses face unique challenges in maintaining robust security measures. Despite the size of your business, you still rely on interconnected networks, systems, and applications to conduct your everyday operations efficiently. So, how will managing machine identity help your small business? Managing your machine identities can help you in numerous ways; here are a few prominent ones. Machine Identity Management helps small businesses to 📍Secure their digital assets 📍Safeguard sensitive data 📍Mitigate risks associated with cyber threats With effective machine identity practices, small business owners can ✅ Improve their overall security posture ✅ Support the growth of their business ✅ Scale their small business But what exactly is machine identity? Understanding Machine Identity Unique cryptographic identifiers used to authenticate and secure communication between machines are called machine identities. These identities are allotted to machines in a networked environment, such as servers, devices, and applications. Managing machine identities is crucial for small business owners to protect against cyber threats and maintain their operational integrity in contemporary IT infrastructures. What Constitutes Machine Identity? Machine identity encompasses various components that establish the digital identity of devices, applications, and services within an IT ecosystem. These include 📍Digital Certificates 📍API Keys 📍SSH (Secure Shell) Keys 📍Device Identifiers Why Is Machine Identity Important For Your Small Business Machine identities are crucial for a secure business environment and serve important functions in the contemporary IT infrastructure. 📌 Authentication: Machine identities help business owners verify the identity of devices, applications, and services, ensuring only trusted entities can access resources and systems. 📌Encryption: Machine identities, especially SSL/TLS certificates, enable encryption of data transmitted over networks, safeguarding it from unauthorized interception and ensuring confidentiality. 📌 Access Control: They have a crucial role in access control by regulating permissions based on machines' identities and roles. This helps prevent unauthorized access and actions within IT environments. 📌Understanding and effectively managing machine identities is essential for maintaining IT operations' security, integrity, and efficiency, especially for small businesses with limited resources and specialized needs. Machine Identity Management Challenges for Small Businesses Unmanaged or improperly machine identities can pose serious challenges to small businesses. Some of the challenges your business can face while managing machine identities. Lack of Awareness and Resources Small businesses often face challenges due to limited awareness of available resources and best practices. This can encompass various aspects: 📍 Technology: Small businesses might not be aware of the latest technologies and tools that can help them streamline their operations and improve security. 📍 Financial: Limited budgets can restrict access to necessary resources such as advanced cybersecurity solutions or professional consulting services. 📍 Training and Education: Owners and employees might lack awareness of cybersecurity risks and protocols, leading to vulnerabilities like phishing attacks or weak password practices. Risks Associated with Unmanaged Machine Identities Machine identities refer to digital certificates and cryptographic keys used to authenticate machines in networks. The risks associated with unmanaged machine identities include: 📍Misconfigured or compromised machine identities can lead to unauthorized access to sensitive data or systems. 📍Improperly managed machine identities can result in compliance failures with industry regulations and standards. 📍Unmanaged machine identities can lead to operational disruptions and downtime. Compliance and Regulatory Concerns While compliance with regulations and standards is crucial for your small business, machine identity management can make it challenging due 📍Various regulatory requirements such as GDPR, HIPAA, and PCI DSS can be complex to understand and adhere to, especially for you as a small business owner. 📍Shortage of dedicated resources to meet compliance requirements is another challenge for small business owners. Hiring a dedicated resource can put a strain on your small business budgets. 📍Non-compliance can result in significant penalties or legal consequences. This is quite a detrimental situation for small businesses with limited financial resilience. Small businesses can address these challenges with strategic resource allocation and leveraging technology solutions tailored to their needs.

Machine Identities in Small Businesses: The Key to Security and Efficiency

Top 100 Places to Learn to Code for Free Online in 2019.

Founders Interview Namahe co-founder Kumar Mudaliar (left), & Adqura Director & Namahe partner Sameer Prakash (right) Disclosure: , the Blockchain-based Supply Chain, has previously sponsored Hacker Noon. Namahe Today we’re going to catch up with Namahe co-founder Kumar Mudaliar , and Adqura Director & Namahe partner Sameer Prakash. Kumar has more than 15 years of experience in the banking sector, specializing in Finance, Strategy and Business Operations. Sameer Prakash led the AI implementation of the platform, and has spent 7 years as CTO of Adqura, Namahe’s partner. Namahe ’s token sale is now live. Learn more at Namahe.io . David : Why does Namahe need to exist? Because even the biggest of companies have not been able to bring in complete transparency in their supply chains and there is still evidence of issues around working conditions, wages, child labour, harassment, and violation of local laws in the supply chains of many industries such as Fashion. Kumar & Sameer: Could you share the company’s founding story? What is the story behind the name of the company, “Namahe”? ‘Namahe’ in sanskrit means ‘thanks to the creator.’ The entire purpose behind is to ensure fair practices in the value creation of a product and fair treatment for each stakeholder involved in creating a product. Namahe The company was founded by Ajay Lakhwani in 2009 based on his experience of managing social responsibility for 9 years at Gap Inc. in South Asia. Ajay was aware of the root causes and the situation on the ground in the formal as well as informal sectors within the apparel supply chain. He has worked on improving working conditions and has partnered with different stakeholders to reduce the ethical issues down the supply chain. However, due to the scale and complexity of these issues, he believed that technology needs to play a big role. Your white paper says Namahe will be the “first socially responsible supply chain.” In layman’s terms, could you explain what that means? If the world had a widely used socially responsible supply chain, what would the social impact be? is a platform that any company in any industry will be able to use to create traceability and transparency in their supply chain. It would join all stakeholders starting from the worker and factories to the retailers and consumers under one platform. In layman’s terms, as a consumer, one could be confident that the products they are using have been manufactured under ethical conditions. As retailers, they would have access to all the necessary information through blockchain, would be able to partner with other retailers to share information and would be able to benefit from efficiencies via Artificial Intelligence. For manufacturers, they would have better control of their manufacturing tiers especially for the operations performed in the informal sector. For the workers, they would be able to feed their own information on the amount of work done, wages received and grievances via the Namahe platform through the user friendly interfaces provided to them. Namahe Is Namahe more of an AI company or a Blockchain company? What type of technology is Namahe? is an amalgamation of both. Namahe The platform provides a true end to end view of the supply chain by using technology to track orders as they get actioned in the spider web that is the modern supply chain. 1. Interfaces The platform provides (mobile & desktop) apps and interfaces to allow registered stakeholders to perform functions on the platform. 2. Blockchain As you would expect, the platform is built on Blockchain with each order a smart contract with expected SLAs and parameters. As the order passes down and then back up the chain, each stakeholder is treated as node in the blockchain and they need to satisfy the requirements of the tasks stated in the contract. Namahe 3. AI AI keeps track of each of the hundreds and thousands of individual tasks and its impacts on the overall delivery SLAs. At the individual level, it is a friendly virtual assistant helping and reminding the users track and complete their tasks. At the platform level it is assessing the impacts and adjusting parts of the chain to compensate and replan to hit targets. This is our solution — a smart Blockchain Platform supported by AI exposed as a series of application UX / UI to the end user. What is the current state of the product? And what technologies will become core to the platform? The concept behind the product was built out in Namahe 1.0. As part of Namahe 2.0 (the current project), we have agreed a real world pilot use case with a prospective brand. The product roadmap looks like this: 1. Design POCs (May-June 2018) As part of this POC we are exploring the available ‘Design’ options on IBM Hyperledger and how these features can be used in the best way on our platform. This is currently underway. 2. Technical Demonstrator (June 2018) In tandem with the Design POC we are building a working version to demonstrate the pilot use case to the prospective brands. This is being kicked off as we speak. 3. First Brand engaged and project kick off (Oct 2018) We are actively working with a top brand and our partners to start the first phase of the live project in Q4–2018. What is Namahe’s short-term and long-term business model? The short-term business model is to divide the end to end supply chain platform into different modules such as the sub-contractor module and the Informal sector module. We are already in discussions with retailers and developmental organisations to test these modules on the ground. Once all modules are completed and tested we would begin licensing it to the retailers and also develop the Namahe Marketplace based around the Namahe platform. This however, would be an independent marketplace enabling small and medium sized manufacturers and retailers to trade. What is the current makeup of the Namahe team (skillsets, location, volume, etc.)? And what will you be hiring for over the next year? (feel free to link job ads) Ajay Lakhwani, Cofounder, UK 17 years expertise in general management, sales and supply chain in e-commerce, fashion and consumer product industries. Diverse experience working in regional and global leadership roles including management of vendor development and social responsibility in South Asia for one of the largest global clothing retailer, Gap Inc. Kumar Mudaliar, Cofounder, UK Has more than 15 years of experience in the banking sector, Kumar specialises in Finance, Strategy and Business Operations. Apart from his accounting and financial repertoire, Kumar is a crypto enthusiast and a believer in the power of Blockchain technology. Kanika Dhyani Ph.D, Data Scientist, Operational Research Specialist, ItalyKanika applies optimisation algorithms and predictive modelling in the ambience of capacity planning in data centres and for HYBRID CLOUD projects including Amazon and Azure. She specialises in Operations Research, Data Analysis, Statistics and Mathematical Modelling. Razvan Pitic Ph.D, RFID, IOT & Wireless Specialist, Italy Experienced technology specialist in the field of IoT/M2M with a strong research background in Wireless communications. Razvan is an expert in IoT and RFID technologies. He specialises in R&D in wireless networks, medium access techniques, resource allocation, packet scheduling algorithms, QoS provisioning and consulting in ICT. Anthony Colón Jr., Blockchain Developer, USA Anthony C Jr is a Blockchain whizkid. He started programming when he was just 14, creating his versions of popular online games. At 17 he went professional, designing Real estate websites but was back in the crypto fold a few years later, as an active member of the ICO community and worked on a number of successful ICOs. He joins Namahe as the lead blockchain developer. Crypto and Music are Anthony’s passion. Sane Lebrun, Head of Marketing & Social Media, France Sane has 8 years of experience working for tech companies in all spheres of digital marketing including brand development, mobile marketing, partnership management, social media growth, SEO, content marketing, influencer marketing and creative asset management. He specialises in designing and driving marketing campaigns for companies within the blockchain space. In the next year, we will be expanding the team and partnering with new retailers and NGOs, working closely within the informal sector as we have with SEWA Bharat. The team will comprise of sales, technical and community team members to deliver the Namahe platform. Why will Namahe be starting in the fashion industry? How will people in fashion adopt it? Namahe will be start with the fashion industry as it’s a $3 Trillion USD industry, has a lengthy and globally spread supply chain, consists of both formal and informal sectors making it non-transparent and therefore more prone to exploitations. It also consists of tens of millions of workers globally, most of whom are from low to very low income groups from under developed markets. People in fashion would adopt Namahe as it would provide them the tools, processes and interfaces that would help them avoid or detect ethical issues in their supply chain, something they have been trying to achieve for decades with minimal success. Moreover, Namahe would use AI along with blockchain for continuous learning and improvements thereby improving efficiencies. When labor exploitations are discovered (such as child labor or below minimum wage labor), how will Namahe intervene? Namahe would provide traceability through the use of blockchain, and would provide processes, tools and interfaces to avoid and detect exploitation. Namahe would be working together with all stakeholders including retailers, manufacturers, government authorities, development organisations and worker groups in an advisory capacity. However, the actions for any issues uncovered would be the responsibility of the relevant stakeholders. Could you tell us about how the NMH coin will work? NMH is a utility token that plays a new and important role in the Namahe Marketplace, where retailers and suppliers, big and small are registered and interact for delivering orders. The Namahe marketplace will be designed as a global platform that will aim to support payments using prevalent digital currencies with preference to NMH tokens (in compliance with local laws and regulation). The ERC-20-compatible NMH token will be traded over the public Ethereum blockchain and will allow users to purchase and sell goods and services on Namahe Marketplace. The NMH token will be listed on major exchanges (to be announced) and will be transferable to other cryptocurrencies and fiat currencies. It is anticipated that demand of the NMH tokens will be driven by trading on the exchanges and the demand for usage on the Namahe marketplace, whilst the number of tokens in the public domain remains fixed. What is the goal (amount and timing) of the token sale? And what discount is currently being offered? The Namahe Token Sale has a soft cap (if the the limit is not reached, the funds would be returned to the investors) of USD 4m and a hard cap of USD 30m. We are issuing 60m NMH tokens for public sale at the rate of 1 ETH = 1000 NMH tokens. Until 6th June we have 20% bonus tokens and thereafter until 20th June we are offering 15% bonus tokens. How can the blockchain create transparency around supply chains in ways that other technologies can’t? The biggest issues with supply chains are the invisible or informal parts of the supply chain, the flow of information and risk of tampering of the records. Using blockchain would help us decentralise the records and make them tamper proof, make the information available to all stakeholders and enable the workers to enter information pertaining to their work. What does Namahe look like in 5 years? In 5 years, would be a platform being widely used by retailers throughout their supply chains. Namahe Marketplace would be creating a buzz as an emerging online place for B2B trade amongst medium to small scale manufacturers and retailers globally. Namahe ’s token sale is now live. Learn more at Namahe Namahe.io

How Blockchain Could Revolutionize the Supply Chain

The Early Days of ArtMap Inc. 

Microservices can help any organization achieve its goal of increasing agility by addressing critical factors such as improving team autonomy, reducing time to market, cost-effectively scaling for load, and avoiding complete outages of the applications. As organizations break their monolith applications into microservices, one of the major hurdles they encounter is identifying database dependencies. Database sharing can be a complex and time-consuming challenge to solve. Databases do not allow you to define what is shared and what is not. While modifying a schema to better serve one microservice, you might inadvertently break how another microservice uses that same database. Additionally, it’s often difficult to identify the data owner and locate the business logic that manipulates the data. In this article, we’ll explore how we might use OpenTelemetry to identify the components that share the same database and database objects, such as tables. Observability and OpenTelemetry: The Basics Before we build our demo application, let’s lay a foundation by discussing observability and OpenTelemetry. What makes an application highly observable? . A system is said to be highly observable if its internal state can be inferred by studying its output at any point in time For example, an observable mobile app that interacts with multiple services can reconstruct a transaction that produces an error response, so that developers can identify the root cause of the failure. An observable application collects three types of information for every transaction: Record of individual events that make up the transaction. Logs: : Record of aggregates of events that make up the transaction. Metrics Record of the latency of operations to identify bottlenecks in the transaction. Traces: What is OpenTelemetry? is a system that generates logs, metrics, and traces in an integrated fashion. OpenTelemetry defines a standard to capture observability data. The OpenTelemetry data model has several key components. OpenTelemetry Attributes Every data structure in OpenTelemetry is comprised of attributes, which are key-value pairs. The OpenTelemetry standard defines what attributes any component (such as an SQL client or an HTTP request) can specify. Events Events are simply a timestamp and a set of attributes. You can record details such as messages and exception details against an event. Context Context comprises attributes that are common to a set of events. These are two types of contexts. (or ) defines the location of events. Their value doesn’t change after the application executable starts. Examples include the name or version of the service or the library name. Static context resource (or ) defines the active operation that contains the event. The value of span attributes changes when the operation executes. Some common examples of span attributes include the start time of a request, the HTTP response status code, or the path of the HTTP request. Dynamic context span In a distributed transaction, the context needs to be passed to all the associated services. In such cases, the receiver service uses the context to produce new spans. A trace that crosses the service boundary becomes a and the process of transferring context to other services is called . distributed trace context propagation Logs Logs are events that only accompany resources. One example of this is the event emitted when a program starts up. Traces Events can be organized into a graph of operations associated with resources. A trace is a graph that presents the events related to a transaction. Metrics An event might occur several times in any application, or its value might change. A metric is an event whose value can be the count of related events or some computation of the value of the event. An example of a metric is the system memory event, its attributes are usage and utilization. To learn about the concepts of OpenTelemetry in detail, refer to the . documentation Using OpenTelemetry to Identify Database Dependencies We discussed earlier that OpenTelemetry prescribes the attributes that the various components of an application should capture. The specification for attributes that should be part of the span covering a database client call is . Many popular languages provide out-of-the-box instrumentation libraries that collect telemetry for database operations. documented on GitHub For this demo, we will use the in coordination with for telemetry storage and analysis. .NET SQLClient instrumentation for OpenTelemetry Lightstep Let’s discuss the architecture of the demo application to understand the path that the telemetry takes to reach Lightstep. We’ll focus our discussion on traces alone, as they are sufficient to identify the dependency between databases and components of a monolith. However, any enterprise application should generate relevant logs and metrics in addition to traces for complete visibility. First, we will instrument our monolith application with the OpenTelemetry SDK to emit observability signals. While instrumenting the application is a manual process for .NET applications, automatic instrumentation is available for applications built with languages such as Golang or Java. We use the OpenTelemetry Protocol (OTLP) Exporter, which is included with the SDK. The exporter lets us send data directly to a telemetry ingestion service. OpenTelemetry platforms such as Jaeger and Lightstep aggregate the traces to help you draw insights. Once integrated with the SDK, the various parts of your application, such as the ASP.NET Core request handler and SQL Client, automatically start producing traces with relevant information. Your code can generate additional traces to enrich the available information. In the case of .NET, the OpenTelemetry implementation is based on the existing types in the namespace as follows: System.Diagnostics.* represents an OpenTelemetry tracer responsible for producing . System.Diagnostics.ActivitySource Spans represents a . System.Diagnostics.Activity Span You can add attributes to a span using the function. Also, you can add baggage using the function. The baggage is transported to the child activities, which might be available in other services using the W3C header. AddTag AddBaggage After instrumenting your application, you can run automated tests or allow users to use your application to cover all the interaction paths between the application and database. Demonstration Let’s create a simple monolithic Employee Management Service (EMS) modeled as an . Our API will have the following endpoints: ASP.NET Core minimal API : Records the hours worked by an employee on a project. POST /ems/billing : Fetches an employee’s hours on different projects. GET /ems/billing/{employeeId} : Adds an employee to payroll. POST /ems/payroll/add : Fetches the payroll data for the employee. GET /ems/payroll/{employeeId} You’ll notice that the monolith serves two distinct domains: billing and payroll. Such dependencies might not be very evident in complex monoliths, and segregating them might require significant code refactoring. However, by studying the dependencies, you can uncouple them with little effort. The complete source code of the EMS application is available in . this GitHub repository Spinning up the database First, we start a SQL server instance in docker: docker run \
-e "ACCEPT_EULA=Y" \
-e "SA_PASSWORD=Str0ngPa$$w0rd" \
-p 1433:1433 \
--name monolith-db \
--hostname sql1 \
-d mcr.microsoft.com/mssql/server:2019-latest We use the following SQL script to create the EMS database and the tables used by our application: IF NOT EXISTS(SELECT * FROM sys.databases WHERE name = 'EMSDb')
BEGIN
  CREATE DATABASE EMSDb
END
GO

USE EMSDb

IF OBJECT_ID('[dbo].[Timekeeping]', 'U') IS NULL
BEGIN
  CREATE TABLE [Timekeeping] (
    [EmployeeId]      INT  NOT NULL,
    [ProjectId]       INT  NOT NULL,
    [WeekClosingDate] DATETIME NOT NULL,
    [HoursWorked]     INT  NOT NULL,
    CONSTRAINT [PK_Timekeeping] PRIMARY KEY CLUSTERED ([EmployeeId] ASC, [ProjectId] ASC,  [WeekClosingDate] ASC)
  )
END
GO

IF OBJECT_ID('[dbo].[Payroll]', 'U') IS NULL
BEGIN
  CREATE TABLE [Payroll] (
    [EmployeeId]   INT   NOT NULL,
    [PayRateInUSD] MONEY DEFAULT 0 NOT NULL,
    CONSTRAINT [PK_Payroll] PRIMARY KEY CLUSTERED ([EmployeeId] ASC)
  )
END
GO Implementing the API service Next, we write the code for the API endpoints. We replace the boilerplate code in the Program class with the following code: var builder = WebApplication.CreateBuilder(args);
builder.Services.AddScoped(_ =>
    new SqlConnection(builder.Configuration.GetConnectionString("EmployeeDbConnectionString")));
var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();

app.MapPost("/ems/billing", async (Timekeeping timekeepingRecord, SqlConnection db) =>
    {
        await db.ExecuteAsync(
            "INSERT INTO Timekeeping Values(@EmployeeId, @ProjectId, @WeekClosingDate, @HoursWorked)",
            timekeepingRecord);
        return Results.Created($"/ems/billing/{timekeepingRecord.EmployeeId}", timekeepingRecord);
    })
    .WithName("RecordProjectWork")
    .Produces(StatusCodes.Status201Created);

app.MapGet("/ems/billing/{empId}/", async (int empId, SqlConnection db) =>
    {
        var result = await db.QueryAsync<Timekeeping>("SELECT * FROM Timekeeping WHERE EmployeeId=@empId", empId);
        return result.Any() ? Results.Ok(result) : Results.NotFound();
    })
    .WithName("GetBillingDetails")
    .Produces<IEnumerable<Timekeeping>>()
    .Produces(StatusCodes.Status404NotFound);

app.MapPost("/ems/payroll/add/", async (Payroll payrollRecord, SqlConnection db) =>
    {
        await db.ExecuteAsync(
            "INSERT INTO Payroll Values(@EmployeeId, @PayRateInUSD)", payrollRecord);
        return Results.Created($"/ems/payroll/{payrollRecord.EmployeeId}", payrollRecord);
    })
    .WithName("AddEmployeeToPayroll")
    .Produces(StatusCodes.Status201Created);

app.MapGet("/ems/payroll/{empId}", async (int empId, SqlConnection db) =>
    {
        var result = await db.QueryAsync<Payroll>("SELECT * FROM Payroll WHERE EmployeeId=@empId", empId);
        return result.Any() ? Results.Ok(result) : Results.NotFound();
    })
    .WithName("GetEmployeePayroll")
    .Produces<IEnumerable<Payroll>>()
    .Produces(StatusCodes.Status404NotFound);

app.Run();


public class Timekeeping
{
    public int EmployeeId { get; set; }
    public int ProjectId { get; set; }
    public DateTime WeekClosingDate { get; set; }
    public int HoursWorked { get; set; }
}

public class Payroll
{
    public int EmployeeId { get; set; }
    public decimal PayRateInUSD { get; set; }
} At this point, we can run the application, test the various endpoints, and look at the records saved in the database. Although the database dependencies of the various endpoints and request paths are clearly evident in this demonstration case, this would not be the case in large applications. Next, let’s automate the process of discovering database dependencies. Adding instrumentation We instrument the application with the and . First, we add the following NuGet package references to the API’s project file: OpenTelemetry SDK SqlClient instrumentation library for. NET <PackageReference Include="OpenTelemetry" Version="1.2.0-rc2" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.2.0-rc2" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.0.0-rc9" /> The SDK provides us with several extension methods that we can use to quickly plug OpenTelemetry into the request processing pipeline. The following piece of code instruments OpenTelemetry in our API. It will also instrument the to emit verbose telemetry. The telemetry from the is key to identifying the database dependencies in detail. SqlClient SqlClient // Configure tracing
builder.Services.AddOpenTelemetryTracing(builder => builder
    // Customize the traces gathered by the HTTP request handler
    .AddAspNetCoreInstrumentation(options =>
    {
        // Only capture the spans generated from the ems/* endpoints
        options.Filter = context => context.Request.Path.Value?.Contains("ems") ?? false;
        options.RecordException = true;
        // Add metadata for the request such as the HTTP method and response length
        options.Enrich = (activity, eventName, rawObject) =>
        {
            switch (eventName)
            {
                case "OnStartActivity":
                {
                    if (rawObject is not HttpRequest httpRequest)
                    {
                        return;
                    }

                    activity.SetTag("requestProtocol", httpRequest.Protocol);
                    activity.SetTag("requestMethod", httpRequest.Method);
                    break;
                }
                case "OnStopActivity":
                {
                    if (rawObject is HttpResponse httpResponse)
                    {
                        activity.SetTag("responseLength", httpResponse.ContentLength);
                    }

                    break;
                }
            }
        };
    })
    // Customize the telemetry generated by the SqlClient
    .AddSqlClientInstrumentation(options =>
    {
        options.EnableConnectionLevelAttributes = true;
        options.SetDbStatementForStoredProcedure = true;
        options.SetDbStatementForText = true;
        options.RecordException = true;
        options.Enrich = (activity, x, y) => activity.SetTag("db.type", "sql");
    })
    .AddSource("my-corp.ems.ems-api")
    // Create resources (key-value pairs) that describe your service such as service name and version
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("ems-api")
        .AddAttributes(new[] { new KeyValuePair<string, object>("service.version", "1.0.0.0") }))
    // Ensures that all activities are recorded and sent to exporter
    .SetSampler(new AlwaysOnSampler())
    // Exports spans to Lightstep
    .AddOtlpExporter(otlpOptions =>
    {
        otlpOptions.Endpoint = new Uri("https://ingest.lightstep.com:443/traces/otlp/v0.9");
        otlpOptions.Headers = $"lightstep-access-token={lsToken}";
        otlpOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
    })); Although the instrumentation is sufficient for us in its current state, let’s enrich the data further by adding relevant traces. First, we define a tracer from which our application spans will originate. var activitySource = new ActivitySource("my-corp.ems.ems-api"); Next, we create a span and add relevant details—attributes and events: app.MapPost("/ems/billing", async (Timekeeping timekeepingRecord, SqlConnection db) =>
    {
        using var activity = activitySource.StartActivity("Record project work", ActivityKind.Server);
        activity?.AddEvent(new ActivityEvent("Project billed"));
        activity?.SetTag(nameof(Timekeeping.EmployeeId), timekeepingRecord.EmployeeId);
        activity?.SetTag(nameof(Timekeeping.ProjectId), timekeepingRecord.ProjectId);
        activity?.SetTag(nameof(Timekeeping.WeekClosingDate), timekeepingRecord.WeekClosingDate);

        await db.ExecuteAsync(
            "INSERT INTO Timekeeping Values(@EmployeeId, @ProjectId, @WeekClosingDate, @HoursWorked)",
            timekeepingRecord);
        return Results.Created($"/ems/billing/{timekeepingRecord.EmployeeId}", timekeepingRecord);
    })
    .WithName("RecordProjectWork")
    .Produces(StatusCodes.Status201Created); We follow the same procedure to instrument the remaining endpoints. Connecting to Lightstep Finally, we need an API key to send traces to Lightstep. We start by . From the Project settings page of our account, we find the Token, which will serve as our API key. creating an account We copy the token and paste it in the file. appsettings {
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "EmployeeDbConnectionString": "Server=localhost;Database=EMSDb;User Id=sa;Password=Str0ngPa$$w0rd;"
  },
  "LsToken": "<Lightstep token>"
} Sending requests Our application is now ready. We launch the application and send some requests to each endpoint. Here is an example of a request that I sent to the endpoint. This request should create a record in the table of the database.' /ems/billing Timekeeping Here’s another request I made to the endpoint to add a record to the Payroll table: /emp/payroll/add When we navigate to the , we can click on the Operations tab to see all the spans Lightstep received from the application. Lightstep observability portal When we click on the operation, we can view the end-to-end trace. By viewing the spans, we can ascertain the sequence of operations for any request. Clicking on the spans brings up its events and attributes, from which we can gain deeper insights into the operation. /ems/payroll/add The final span visible in the trace is , which was generated by our instrumented SQL Client. We click on the span to view its attributes and events as follows: EMSDb We can draw some key insights from the attributes: The name of the database The SQL statement used in the database operation The type of SQL statement (text or stored procedure) The hostname of the service that made the request We find a similar set of details from the child span of the /ems/billing operation. By combing the information from the traces, we can infer the following: Ingress operations (operations that receive the external request) The sequence of activities to fulfill a request, including external service calls and database operations. Database operations involved in each operation. Together, this information is sufficient for us to plan the segregation of services and databases and establish contacts for communication between the microservices. Conclusion This article discussed one of the common challenges that developers encounter while transitioning their monolith applications to microservices. Of all the concerns, splitting a database is a complex endeavor because any service that has access to the database can manipulate it. By using OpenTelemetry, we can identify the dependencies among the various components and between the components and the database. With the knowledge and understanding of our dependencies, we can develop the refactoring plan for our components, planning how they should evolve as independent microservices.

How to Use OpenTelemetry to Identify Database Dependencies

As time goes on, more and more exceptional females are making their mark in the technology world and inspiring many others to follow suit. The North of England is becoming a vibrant region of digital innovation, with a large percentage of the people truly enhancing the digital sector, being female. In this blog we recognise just some of the Female Leaders making a difference in the tech community in the North. Zandra Moore is the CEO of award winning software company — (based in Leeds). Zandra Panintelligence Zandra is passionate about empowering women in tech and inspiring more women into the industry. She founded the and gives regular talks on gender diversity and women in tech. Zandra was also recently shortlisted at the in the “Rising Star”category. Leeds Lean in Ladies Circle Forward Ladies Awards Zandra’s work with Panintelligence and her proactivity in the community, driving growth in the sector, is making her one of the most inspiring female leaders in tech, in the North, at the moment. Melissa Hendry Melissa is Operations Director at two start-ups ( and ) in the North and has recently been shortlisted at this year’s Forward Ladies Awards and Summit in the STEM Rising Star category. ZiMovi VST Enterpises Melissa is extremely passionate about driving further development in the tech sector in Leeds. She said in a recent Q&A with us that “Leeds has an incredibly thriving tech scene and most importantly community. There is an amazing number of inspiring and educational, free events held each week in the city, which is incredible. Assisting people to widen their connections, learn from their peers and trailblazing companies and offer environments to forge collaborations is vital. Leeds also have some brilliant initiatives in promoting and nurturing STEM subjects and courses at primary, secondary and higher education; from summer schools, placements, gap-years, workplace outreach and apprenticeships.” Deb Hetherington Deb’s work with impacts the digital sector in the North in a range of ways, from sizeable knowledge transfer partnerships with large scale projects, right through to working with digital start-up companies offering the full spectrum of university services. Leeds Beckett University Deb’s main project is a European funded programme , which offers digital start-up companies assistance in marketing, business planning, app development, mentoring, workshops, funding and more. Deb is able to meet with companies interested in the programme and run through a full diagnostic of their business in order to gauge where they are now, where they are aiming to be, and what their barriers to growth are. She can then create a business support programme for them made up of the services listed above. called Ad:venture Here’s what Deb has to say about the exciting digital landscape in The North at the moment “The digital sector in the North is a really exciting place to be at the moment, particularly in Leeds. There is so much going on, and the best part is how collaborative we are as a city — with such collaboration and partnership potential the opportunities are endless. It’s also great to see so many female leaders in tech here, people who are flying the flag for diversity and are equally bloody good at their jobs; Zandra Moore CEO of Panintelligence, Natasha Sayce-Zelem head of tech at Sky, Tanja Lichtensteiger Engineering Manager of Sky Bet to name but a few. There is a strong network of women in tech in the North, who also provide mentorship for young women coming through education who want to work in digital. Educating the next generation is hugely important in order to maintain our position on the digital map and retain the talent our university’s are churning out.“ Anna Sutton Anna Sutton set up the in 2014 alongside her husband Ed. Data Shed collates and analyses data from a wide range of sources to give its clients a better insight into the needs and expectations of their customers. The idea stemmed from Anna’s frustration at getting insight from data within the organisations she was was working for. Anna and Ed discussed and decided they wanted to build a product that could help non-technical people process and analyse their own data sources. Their determination to be self-funded meant they had to find money from somewhere and so the consultancy was born! The Data Shed Here’s what she has to say about running her own business and being a female leader, “There’s never a dull day running your own business. Learning to lead “on the job” in a totally new industry has been an immense challenge (particularly juggling alongside motherhood). Most fulfilling has been building a brilliant team and making sure we have an environment that lets us all thrive.” Anna is active in the Leeds Digital community and can often be found speaking at regular local events including Ladies of Code and was recently on the Panel of the Leeds event of the Tech Nation tour. Natasha Sayce-Zelem is Head of Technology at Sky and has been named one of the UK’s Top 100 Women working in technology right now. Natasha Sayce-Zelem Natasha is also the founder of the movement in Leeds. The movement is working to showcase inspiring role models working in Science, digital and Technology, in a way to support and attract more women in the industry. Empowering Women with Science and Tech Tanja Lichtensteiger is Engineering Manager at in Leeds. Tanja has over 17 years’ experience in creating systems. She is active an member of the Leeds tech community and is proactive in championing women in tech across the UK. Tanja is featured on T and was awarded the FDM Every Women in Technology Award in 2018! Tanja Sky Betting and Gaming he Northern Power Women ‘Future List 2018’ Debbie Ashton (Pic: Financial Force) Senior Vice President of strategic customer experience and a founder of FinancialForce.com. Debbie has been instrumental in the growth and ongoing success of the FinancialForce.com business. Here’s what Debbie has to say about being a Female Tech Leader, “As the co-founder of FinancialForce I have been incredibly lucky to have the opportunity to build a business where the culture, attitude and values reflect what I feel are most important to give our teams and customer the best experience possible. We have fantastic flexibility with which our employees are able to balance their personal lives alongside their working life, super employee benefits that are designed specifically to give options around what you want to spend your allowances on and finally a culture that is diverse, totally inclusive and understanding to the individuals needs. As a female leader I am incredibly proud to have nurtured this environment as it runs through our core DNA as a business. I’m very proud of our culture and believe we provide a fantastic opportunity for future generations in the tech industry, Financial Force is an inclusive place for everyone to work.” Catherine Trotter (Pic: Manchester Evening News) Catherine is Technology Director at Group. She has been mentioned in the as being one of the top 30 Women in Tech in Manchester earlier this year. MoneySupermarket Manchester Evening News In a recent interview with , Catherine told readers how her father originally sparked her interest in technology. Laudale “He was an electronic engineer and was forever bringing home gadgets and things for us to try, we had circuit boards knocking around our house from when I was 4 years old so I think my Dad is really responsible for my interest in technology and he was the one who encouraged me to study technology at university when I didn’t get the grades to do the course I wanted!” Helen Oldham Helen is a founding director at A Yorkshire based (not for profit) investor platform which is accelerating the growth of tech start-ups, and other tech businesses in FinTech, Health Tech, media and Tech for good sectors. NorthInvest. Helen and North Invest are giving tech businesses the support to reach their goals and full potential. Chloe Barrett (Pic: Bdaily) Ed Tech entrepreneur Chloe is founder of Digi Dentistry. Digi Dentistry is a visual learning platform which is an advanced way of learning using augmented reality and animation targeting all learning domains. Enhancing and changing dental education for the better. Chloe’s inspiring work at DigiDentistry found her winning this year’s ‘ at this year’s Great British Entrepeneur Awards. Chloe is also a finalist at the Digital Entrepreneur Awards and UK Tech Founder Awards! Young Entrepeneur of The Year’ Here’s what Chloe had to say about being a female tech leader, “I have been able to create something with technology which will ultimately change the way we educate the current and next generation of learners. I feel as a female tech founder this is life changing, having the opportunity to inspire other women to feel encouraged to start a company of their own, demonstrating we all have to start somewhere and build on our experiences.” Amy De Balsi (Pic: Bradford Telegraph) launched in 2015. Herd specialises in pulling together the best digital and tech vacancies from local employers in Leeds and then pushing them out to potential candidates on social media. Amy De Balsi Herd digital job board Amy also founded the . The job fair brings together an array of local tech employers in the First Direct Arena for an afternoon. The event allows attendees to hear companies pitching about how great it is to work for them and at the last event there were over 500 jobs available. Leeds Digital Job fair The event has seen thousands of people visit over its five events. The fair has been a great showcase of the sheer amount of opportunity for aspiring tech professionals in the region! Lou Cordwell (Pic: Manchester Evening News) is CEO of , an award winning independent digital design company based in Manchester. Lou Magnetic North Alongside her day job she’s also a board member of greater Manchester’s local enterprise partnership, chair of and director of The North. Design Manchester AllBright’s Lou was awarded an for her services to the creative and digital economy. OBE in 2018 New Years honours Fabiola Detari is the Founder and CEO of Engagify Ltd. Fabiola Fabiola has recognised the millions of people globally struggling with mental health and the lack of psychologists to take care of everyone. Fabiola and her team are currently developing an intricate AI solution to support employee mental health 24/7 on a global scale — Sayge (coming soon). Fabiola says, “The tech space offers less degree of prejudice to anyone regardless of age, sex or background. We need more visionaries ‘Ada Lovelace’s’ in the world to create products never considered before now, but once made, could not imagine living without!” Emma Cheshire (Pic: DotForge) Emma is co-founder of FutureLabs. offer companies a number of events and services including, talks and networking. Promoting collaboration, support and ultimately bringing the tech community in Leeds together. Futurelabs Alongside this, Emma is also CEO of . DotForge is a 3 month accelerator programme based in Leeds, Machester and Sheffield taking Tech4Good startups on a supportive journey from ideation, to launch and beyond! DotForge Helen Wollaston Helen is the Chief Executive of the The WISE campaign enables and energises business, industry and education to increase participation and success to Women in STEM. WISE Campaign. Since becoming Chief Executive in 2012, Helen has transformed WISE and has continued to drive the campaign further. Helen is an advocate for social change, with a track record of transformational leadership across public, business and not-for-profit sectors. She founded her own consultancy company, Equal to the Occasion, after seven years as Director of Campaigns for the Equal Opportunities Commission and previously ran the Yorkshire region for the National Lottery Charities Board. Hazel Jones (Pic: Yorkshire Evening Post) Hazel is Programme Director for Apps and Wearables at NHS Digital in Leeds. Hazel and her talented team are behind some of the most iconic innovation taking place at the NHS at the moment. The NHS Apps Library was launched as a beta site in April 2017 and provides a library of digital health tools which are accessible to patients, trusted and easy to use and help them make better choices about digital health and care. Hazel says “The Apps Library has already proved to be a great success. There have been over a quarter of a million visits in the last year and more than half of those were from mobile phones, which shows how patient access to health care is transforming.” Anna Barsby (Pic: CIO) Anna Barsby is CTO at . Morrisons has been focusing on the technology opportunities which provide business growth, and has been encouraging key skills and people to the supermarket retailer as part of her ambitious improvement plans. Anna Barsby The 2015 100 leader and her team are initiating major moves into cloud, digital and data — acutely aware of the need for the supermarket’s back-end systems and infrastructure to support its front-end, customer facing touchpoints. CIO — which has included having Morrisons developers offer training for the coding club at a local girls’ school — Barsby has set up a diversity and inclusion group in her technology function, an initiative she is now sponsoring across the whole business. As well as being active in schools locally Naomi Timperley (Pic: Digital Leaders) is an Honorary Industry Fellow at the University of Salford Business School, co-founder of and chair of investment platform Capital Pilot. Naomi Timperley Tech North Advocates She was named as one of the — a definite list of entrepreneurs who support the entrepreneurs of tomorrow from the Centre of Entrepreneurs. Naomi is also a mentor and was nominated in 2015 by the European Mentoring and Coaching Council for Mentor of the Year. Maserati 100 in 2015 She is a TEDx speaker and an experienced partnerships and engagement consultant with established relationships across the UK — specialising in the tech and digital sector. Naomi was named as a Rising Star in the Computer Weekly Most Influential Women in UK IT 2017 and sits on the board of Future Everything. Dorne Mcloughin Dorne is the CEO/Founder of Ribbit Media Solutions. Her agency specialises in Facebook Advertising, Mobile Wallet Loyalty Solutions and Messenger Chatbot Marketing for local serviced-based businesses, including franchises. In 2016 Dorne transitioned Ribbit Media into the advertising agency that it is today, but prior to this she has trained over 300 business owners in Facebook for Business Workshops in her local town of Halifax, as well as building an online training program. In September of this year she was asked to be one of the speakers at the largest Messenger Chatbot Marketing conferences in the world in Texas, where she spoke on stage to over 1,000 agency owners and marketers. We asked Dorne what excites her the most about having a digital advertising agency and she had this to say…. “one example would be — a client of ours had worries about their lack of wedding bookings for 2019 so we built them a Messenger Chatbot Marketing funnel and combined it with Facebook Advertising. In less than 3-months they took £11,000 in wedding deposits, with total sales valued at approx. £100,000 and with a Facebook Ad Spend of just £1,600 — this is what excites me! With our digital skills we’re able to transform the lives of others and that’s a nice feeling to have”. Gemma Cameron Gemma is principal software engineer at in Manchester. Co-Op Digital Gemma has been referred to by Tech Nation as the “North West tech community Dynamo”. Gemma’s love for tech started early at 8 years old when her father brought home a laptop. She then went on to study computer science at university. Gemma’s commitment to the tech community is admirable. Monday to Friday she is making great things happen at Co-op digital, building their engineering community. Her work outside of co-op digital has made her a key part of the north west tech scene. Gemma is a founder of Hack Manchester, a non for profit events organising company that showcases and benefits the tech community by promoting, celebrating and teaching technical skills through workshops and events for all ages. Professor Janet Cade Janet runs the Nutritional Epidemiology Group at the University of Leeds. Janet is the mind behind . A pioneering online technology helping researchers, teaching and health professionals measure food and nutrient intake. myfood24 Janet Says, _“The frustrations of working with out-dated methods to measure food and nutrient intake challenged me to develop a new approach (Myfood24) using online technology; harnessing the power of data to deliver high quality, evidence based results. The need for such a system led to the foundation, in 2017, of Dietary Assessment Ltd, of which I am founder and Directo_r. I have a strong direction and vision for the company, where we see a critical need to provide a system to the NHS (and other health services across the world) which allows patients to record their food intake to generate nutrient intake results which can be shared with their dietitian or doctor/nurse. We have demonstrated that, not only do we have a validated system for tracking diet, but also that this can lead to better health outcomes for patients and is supported by health care professionals.” originally published here.

21 FEMALE TECH LEADERS FLYING THE FLAG FOR NORTHERN TECH IN THE UK

is a popular technology for efficient and scalable data storage and retrieval. However, maintaining its performance and data integrity requires a crucial practice called reindexing. Indexing is the initial process of adding data to Elasticsearch, while reindexing is essential for maintaining data accuracy and optimizing search performance. Elasticsearch Whether you're a seasoned Elasticsearch user or just beginning your journey, understanding reindexing is important for maintaining an efficient Elasticsearch cluster. In this article, we'll delve into the essentials of Elasticsearch reindexing, answering when it's necessary, how to trigger it, and the best practices to get the most out of your Elasticsearch cluster. Understanding Elasticsearch Reindexing In Elasticsearch, reindexing helps maintain data integrity and increase performance. Put simply, it's the process of copying data from one index to another. While this might sound straightforward, if not done correctly it can cause issues such as slow data retrieval or even incorrect results Imagine your Elasticsearch indices as well-organized libraries. Over time, books might need to be updated, rearranged, or even replaced. Reindexing is akin to rearranging the library shelves or updating the books to keep everything in order. Without it, your library can become disorganized, leading to slower searches and potential inaccuracies in your data. This analogy underscores the importance of understanding reindexing in Elasticsearch. It's not just about copying data; it's about maintaining the integrity of your "library" for efficient searching and retrieval. Let’s take a look at and how to keep on top of it. when reindexing is required When is Reindexing Necessary? Reindexing becomes essential when changes occur in your Elasticsearch data models or mappings, or when you're seeking performance enhancements. In this section, we’ll look into these scenarios in more detail to understand the nuances around why reindexing is required. Structural Changes in Data Models Structural changes in data models refer to modifications in how data is structured within Elasticsearch. These changes can include things like adding or removing new fields or altering data types of existing fields. Introducing new fields often requires a reindex to ensure Elasticsearch knows how to efficiently search for data stored in that field. Modifying data types requires a new index altogether as you cannot change data types in place. Once the new mapping has been created for the modified data type then the data needs reindexing. These structural changes require reindexing due to Elasticsearch's schema-on-write approach. Elasticsearch indexes data as it is ingested, and any changes to the data structure can lead to inconsistencies between existing data and data written with the new schema. As a result, without reindexing, search queries may yield unexpected or inaccurate results due to the schema mismatch of data items. This can have an impact on both data accuracy and search performance. Mapping Updates or Changes Mappings serve as the blueprint for how data is indexed and queried in Elasticsearch. When these mappings are modified then reindexing is usually required. Mappings define the data types and properties of fields within Elasticsearch. Any change to these mappings affects how data is indexed, stored, and retrieved. For instance, altering a text field to a date field fundamentally changes how data is processed and queried. Elasticsearch enforces data consistency based on mapping definitions. Changes to mappings can lead to inconsistencies between existing data and the updated schema if the data is not reindexed. When mappings are modified, particularly if it involves changing data types or field properties, backfilling also becomes important. Backfilling is the process of retroactively populating or updating existing data to align it with a new schema or data structure. This means that the existing data can still be queried efficiently and accurately after the mapping change. Performance Enhancements and Index Optimizations Reindexing isn't just a routine maintenance task, it's a powerful tool for optimizing search performance within Elasticsearch. For example, reindexing allows you to modify the number of shards in an index. Adjusting the shard count, or resharding, can distribute data more evenly, preventing uneven workloads on specific nodes to improve search performance. Reindexing can also be used to consolidate indices together. Let's say you have multiple small indices that share the same data structure and are frequently queried together. Reindexing can consolidate them into a single, larger index. This reduces the overhead of managing numerous small indices which can in turn enhance search speed. Finally, reindexing can be used to improve routing. By reindexing and applying routing strategies effectively, you can route queries to specific shards, minimizing the number of shards that need to be searched. This targeted approach can significantly speed up search queries if your data is frequently searched by specific keys such as a user ID. Upgrading Your Cluster When upgrading from Elasticsearch version 6.X to 8.0 (current major version) and beyond, you may need to reindex any indices that were created in version 6. Elasticsearch's data structures and underlying mechanisms changed significantly between these versions requiring reindexing for compatibility and optimal performance. The reindexing process ensures that data aligns with the updated structure and new functionality to ensure you can migrate seamlessly from old to new. Elasticsearch recommends using their upgrade assistant to help with this process. How to Trigger a Reindexing Operation Reindexing in Elasticsearch is made possible through the Elasticsearch Reindex API. The Reindex API serves as the bridge between your existing index and the new index you want to create or modify. Its primary purpose is to enable the efficient transfer of data from one index to another, on top of this, you can also: Selectively copy documents from the source index to the target index. Apply complex data transformations, such as field renaming or type conversions. Filter data based on specific criteria. Control the indexing process with options like throttling and refresh intervals. Before using the Reindex API, ensure that the target index, where you want to move or transform your data, is created and properly configured. To trigger reindexing, you then need to formulate a POST request to the endpoint, specifying the source and target indices, as well as any desired transformations or filters. An example reindex POST request could look as follows. _reindex POST /_reindex
{
  "source": {
    "index": "source_index"
  },
  "dest": {
    "index": "target_index"
  },
  "script": {
    "source": "ctx._source.new_field = 'transformed value'"
  },
  "query": {
    "term": {
      "category.keyword": "example"
    }
  }
} Once your request is built you can send the request to Elasticsearch, initiating the reindexing process. Elasticsearch will start copying data from the source index to the target index, following your defined instructions. Once the reindexing is complete, thoroughly test the data in the target index to ensure it aligns with your expectations. For example, you can compare the field mappings between the source and target indices to confirm that fields were mapped correctly during reindexing. You could also retrieve a sample of documents from both the source and target indices and compare them to verify the data was reindexed accurately. Best Practices for Reindexing When reindexing within Elasticsearch, you should look to follow these best practices to ensure the reindexing procedure is smooth with no data loss and little impact on existing cluster operations. Prioritize Data Backup Before initiating any reindexing activity, it is important to back up your cluster. This precautionary step acts as a safety net, offering a way to revert to the original state should any unexpected issues arise during the reindexing process. The source index should still exist after reindexing, however, it's a fundamental principle to always have a reliable copy of your data before making significant changes. Conduct Reindexing in a Controlled Environment First To mitigate potential risks and challenges during reindexing, it is advisable to perform the operation in a pre-production environment first. By doing so, you can identify and address any unforeseen issues without affecting the production system. Once the procedure has been completed and verified in the pre-production environment, it can then safely be run in production. Monitor Resource Usage It is important to monitor system resources during reindexing to prevent strain on your infrastructure. Reindexing can be resource-intensive, especially for larger datasets. Keeping a close eye on CPU, memory, disk usage, and network activity can help optimize resource allocation, ensuring the process runs efficiently without causing performance bottlenecks. To check resource usage you can use the node stats API. GET /_nodes/stats This will return a response that looks as follows. {
  "_nodes": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "cluster_name": "my_cluster",
  "nodes": {
    "node_id1": {
      "name": "node_name1",
      "process": {
        "cpu": {
          "percent": 30,
        }
      },
      "jvm": {
        "mem": {
          "heap_used_percent": 40.3,
          "heap_used_in_bytes": 123456789,
          "heap_max_in_bytes": 256000000
        }
      }
    },
    "node_id2": {
      "name": "node_name2",
      "process": {
        "cpu": {
          "percent": 50,
        }
      },
      "jvm": {
        "mem": {
          "heap_used_percent": 60.8,
          "heap_used_in_bytes": 210987654,
          "heap_max_in_bytes": 256000000
        }
      }
    }
  }
} If you find reindexing is too intensive, you can throttle the process by setting the parameter when submitting the reindex request. This will add a sleep between batches for the number of seconds set by the parameter, to provide a cooldown period between batches. requests_per_second Verify and Validate Results Once the reindexing is complete you should verify the data in the target index to ensure it looks as expected. This validation process should encompass a variety of tests including document counts, field mappings, and search queries. Alternative Solutions Elasticsearch has undoubtedly established itself as a prominent solution in the NoSQL search and analytics space. However, it's worth exploring alternative solutions that offer unique approaches to data indexing and querying, particularly one like Rockset. Rockset is a to Elasticsearch and offers a different perspective on indexing and querying data. Unlike Elasticsearch's schema-on-write approach, Rockset allows schemaless ingestion. Data can be ingested and queried without the need for upfront schema definition, offering more flexibility in handling ever-evolving datasets without the need for reindexing. cloud-native alternative In the area of index management, Rockset benefits from its model where a row index, a column index, and a search index are all created automatically for the data as it is ingested. This contrasts with Elasticsearch, where indexes are created by users and structural changes often necessitate time-consuming reindexing procedures. converged indexing While Elasticsearch remains a robust solution for various use cases, exploring alternatives like Rockset may be useful, especially if you find reindexing in Elasticsearch becoming a frequent activity. Conclusion Reindexing is a fundamental process in Elasticsearch and is important for maintaining the efficiency and accuracy of search results as data structures evolve. If you find that reindexing is becoming a constant time burden for your team it might be worth exploring alternative solutions like Rockset. Rockset offers a more streamlined index management process that enables developers to concentrate on more value-add activities.

ARCSOURCE CONSULTING

arcsource.com

11-50 employees

Since 1998

COMPANY RANKING

ARCSOURCE CONSULTING

EVERGREEN INDEX #811

Representing this company? Click here to claim and customize this page!

ArcSource Consulting WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

ARCSOURCE CONSULTING

arcsource.com

11-50 employees

Since 1998

COMPANY RANKING

ARCSOURCE CONSULTING

EVERGREEN INDEX #811

RELATED COMPANIESRANK

STORIES ABOUT ARCSOURCE CONSULTING

Representing this company? Click here to claim and customize this page!

ArcSource Consulting WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!