Moneytor is leveraging technology to make the currently fragmented stressed receivables market more efficient and improve default recovery rates utilizing machine learning and proprietary algorithms

Moneytor

It happens every few months: an article or video describing a shocking new TikTok trend lands in one of your feeds, one you simply can not believe is real. From people biting into Tide detergent pods and trying to climb across stacked crates to eating corn on the cob attached to a power drill, these trends become a core part of the cultural discourse for a few days or weeks before the world turns and moves on to the next one. As TikTok is fundamentally an entertainment vehicle, it’s not surprising its users seek this sort of thing out, nor is it surprising the algorithm built to capture your attention rewards these content creators. What those who don’t use TikTok may find surprising is there is increasing visibility and popularity of edutainment communities, people who gather around content subjects designed to entertain and also educate. There is an untold number of these communities, including #BookTok, #FoodTok, #MoneyTok, and #BeautyTok. But there’s one that has really lifted off in the last year or two that no one saw coming: #PrisonTok. #PrisonTok is an Eye-Opener Like many niche communities on TikTok, #PrisonTok offers a window into a world many know only through the lens of mass media and news reports. #PrisonTok creators come from all angles, including legal professionals, those who work in prisons, the formerly incarcerated, and even, believe it or not, . those who are currently serving time and accessing the internet with a smartphone they’re not allowed to have One leading #PrisonTok influencer, Sabrena Morgan, has built an engaged audience with more than 51,000 followers as she shares her journey from addiction and self-destruction to her four years in federal prison and where she is now, re-entering society while trying to spark a broader conversation about the justice-related things people tend to shy away from. In a recent episode of the , Sabrena shares the emotional evolution she underwent while on pre-trial and away at prison and how that helped her not only develop good, healthy habits but also to live a life driven by intention and a desire to, in her words, “not waste time.” Nightmare Success podcast with Brent Cassity The honest, raw perspective of someone who re-invents their fundamental approach to life is not one that makes its way through to Hollywood, but its popularity on TikTok tells us that it is a mistake by the major studios and producers: people want these stories. Beyond Entertainment and Human Interest And while other #PrisonTok influencers take a sillier approach to their content, offering up things like prison recipes, it’s important to understand the real value of #PrisonTok extends beyond entertainment and human interest. In the words of , #PrisonTok and related communities have become an important step for those facing the prospect of incarceration: “Unfortunately, for many years, there has been a lack of information to those who are going to prison. That’s a very big problem that social media is beginning to address, and the availability of this information has already had a meaningful impact on thousands of people.” federal prison consultant Sam Mangel Mangel, who offers similar, , knows first-hand. After he was arrested for insurance fraud violations in 2016, Mangel engaged an attorney who specializes in white-collar criminal defense. Despite his purported expertise, Mangel’s lawyer failed to prepare him for several key pre-sentencing steps, which culminated in a sixty-month sentence, more than three times greater than the guideline recommendation for his crime. complementary, and informative content about prison on his YouTube channel Mangel turned his attention to what he could do to return to his family as quickly as possible, which was a tall order in the pre-#PrisonTok world. Mangel ultimately served less than two years of his sixty-month sentence, returned home, and now assists white-collar offenders (including CEOs, medical professionals, politicians, and foreign nationals) in preparing for incarceration as a federal prison consultant. If he were faced with the same circumstances in today’s information environment, it’s likely Mangel would have been better informed and would have known to speak with his attorney about the critical steps they missed. It’s possible he would have engaged the services of a consultant for personalized guidance. And armed with all of that, it’s likely he would have found himself with a far better outcome than a sixty-month sentence. The Power of Breaking Down Information Silos While many media outlets focus on security concerns that threaten to and groups that liken social media use to smoking cigarettes, communities like #PrisonTok offer what has long been the promise of social media and the internet broadly: breaking down data silos, improving access to critical information, and connecting people. That’s why it’s important to highlight communities like these, which clearly offer so much help to those in need. These are an important counter-point to the current discourse around TikTok and social media. ban TikTok in the United States

The Rise (and Real Value) of #PrisonTok

#PrisonTok-এর উত্থান (এবং আসল মূল্য)

Der Aufstieg (und der wahre Wert) von #PrisonTok

El auge (y el valor real) de #PrisonTok

L'essor (et la valeur réelle) de #PrisonTok

#PrisonTok का उदय (और वास्तविक मूल्य)

#PrisonTok の台頭（そして真の価値）

#PrisonTok의 상승(및 실제 가치)

A ascensão (e valor real) de #PrisonTok

Рост (и реальная ценность) #PrisonTok

#PrisonTok'un Yükselişi (ve Gerçek Değeri)

Sự trỗi dậy (và giá trị thực) của #PrisonTok

#PrisonTok 的兴起（及其真正价值）

86 Stories To Learn About Payments

Real-Time Event Monitoring with Redpanda, OpenTelemetry, and Grafana

Arize AI, an AI observability and large language model (LLM) evaluation platform, launched prompt variable monitoring and analysis onstage at Google Cloud Next '24 this week. The debut comes at a time of great need. Although enterprises are racing to deploy foundation models to stay competitive in an increasingly AI-driven world, hallucinations and accuracy of responses remain barriers to production deployments. Arize’s new prompt variable monitoring helps AI engineers and machine learning teams automatically detect bugs in prompt variables and pinpoint problematic datasets. Through introspection and refinement of the prompts used in LLM-powered applications, teams can ensure that generated outputs align with expectations around metrics such as accuracy, relevance, and correctness. Additional context window management tools also launching today allow for further examination. “Debugging LLM systems is far too painful today,” says Jason Lopatecki, CEO and Co-Founder of Arize AI. “By analyzing how AI systems respond to a myriad of prompts and offering deeper insights into model behavior, Arize’s new prompt variable analysis tools promise to help AI engineers have more successful outcomes in production — with training and feedback loops to inform ongoing refinement.” What is Arize? Arize AI is an AI observability and LLM evaluation platform that helps teams deliver and maintain more successful AI in production. Arize’s automated monitoring and observability platform allows teams to quickly detect issues when they emerge, troubleshoot why they happened, and improve overall performance across both traditional ML and generative use cases like RAG LLM and query engine for text-to-SQL. Arize is headquartered in Berkeley, CA. Arize AI , an AI observability and large language model (LLM) evaluation platform, launched prompt variable monitoring and analysis onstage at Google Cloud Next '24 this week. Arize AI The debut comes at a time of great need. Although enterprises are racing to deploy foundation models to stay competitive in an increasingly AI-driven world, hallucinations and accuracy of responses remain barriers to production deployments. Arize’s new prompt variable monitoring helps AI engineers and machine learning teams automatically detect bugs in prompt variables and pinpoint problematic datasets. Through introspection and refinement of the prompts used in LLM-powered applications, teams can ensure that generated outputs align with expectations around metrics such as accuracy, relevance, and correctness. Additional context window management tools also launching today allow for further examination. “Debugging LLM systems is far too painful today,” says Jason Lopatecki, CEO and Co-Founder of Arize AI. “By analyzing how AI systems respond to a myriad of prompts and offering deeper insights into model behavior, Arize’s new prompt variable analysis tools promise to help AI engineers have more successful outcomes in production — with training and feedback loops to inform ongoing refinement.” What is Arize? Arize AI is an AI observability and LLM evaluation platform that helps teams deliver and maintain more successful AI in production. Arize’s automated monitoring and observability platform allows teams to quickly detect issues when they emerge, troubleshoot why they happened, and improve overall performance across both traditional ML and generative use cases like RAG LLM and query engine for text-to-SQL. Arize is headquartered in Berkeley, CA. RAG LLM query engine

Arize AI Leads the Way in AI Observability with Prompt Variable Monitoring

Arize AI প্রম্পট ভেরিয়েবল মনিটরিং সহ AI পর্যবেক্ষণযোগ্যতার পথ দেখায়

Arize AI ist mit Prompt Variable Monitoring führend in der KI-Beobachtung

Arize AI lidera el camino en la observabilidad de la IA con monitoreo rápido de variables

Arize AI ouvre la voie en matière d'observabilité de l'IA avec une surveillance rapide des variables

एराइज़ एआई त्वरित परिवर्तनशील निगरानी के साथ एआई अवलोकनशीलता में अग्रणी है

Arize AI は、迅速な変数モニタリングにより AI の観測可能性をリードしています

Arize AI는 신속한 변수 모니터링을 통해 AI 관찰 가능성을 선도합니다.

Arize AI lidera o caminho em observabilidade de IA com monitoramento imediato de variáveis

Arize AI лидирует в области наблюдения за искусственным интеллектом благодаря оперативному мониторингу переменных

Arize AI, Hızlı Değişken İzleme ile Yapay Zeka Gözlemlenebilirliğinde Öncülük Ediyor

Arize AI dẫn đầu về khả năng quan sát AI với tính năng giám sát biến đổi nhanh chóng

Arize AI 凭借快速变量监测引领 AI 可观测性

is a distributed unit of infrastructure. It can provide information about FreeBSD and Linux operating systems. It also supports metrics transmission to Clickhouse, InfluxDB, and Prometheus. Alligator The main interface of alligator is . It connects with spreading Prometheus as a monitoring system in many infrastructures. However, the Prometheus ecosystem has some problems. OpenMetrics First of all - the specialized and simplified agents. Every service on the host needs a specialized exporter. This is the reason for having a lot of exporters per host (like a node_exporter, redis_exporter, sentinel_exporter, and so on). Secondly, simplified agents. A lot of agents can be used for different tasks (like an SNMP, blackbox, etc.), and it inflates infrastructure for such daemons to collect information. Sometimes, a pull model has problems with multi-worker applications and cron jobs - nothing can give metrics from hosts when Prometheus makes a request, or it is connected with problems related to synchronizing results between workers. It forces us to use statsd/pushgateway. Then the question comes up: how to remove outdated metrics? How to make it a high-availability service? The approach of the alligator is different - it is a smart agent for the Prometheus. Alligator consists of a system exporter and an application exporter. It also provides a push model for pushgateway, statsd, and graphite emulation protocol. Metrics Export Alligator supports such sets of metrics: FreeBSD and GNU/Linux system metrics. Filesystem metrics (file’s checksum, sizes, and access level). Certificate metrics (PEM and P12). Some popular applications (ElasticSearch, uwsgi, Druid, RabbitMQ, and others). Ephemeral metrics. It supports pushgateway, statsd, and graphite protocol, and it can collect metrics from file on FS. It also has a cluster mode for centralized installation of pushgateway. Simple JSON to metrics parser. Simple HTTP/TCP checker. Use Cases A part of supported technologies for collecting metrics are reviewed in the . The rest I am describing here. official documentation Application Server Metrics First of all, alligator can collect application server metrics. For instance, alligator can gather metrics from uwsgi instances with this configuration: aggregate {
    uwsgi tcp://localhost:1717;
} and rerun the uwsgi server with the : expose stats port --stats 127.0.0.1:1717 Another example, fastcgi-application: aggregate {
    php-fpm fastcgi://127.0.0.1:9000/stats?json&full;
} and write this option in : php-fpm config pm.status_path = /stats After this, you will have common metrics from the application server. Application Metrics I recommend using synchronization structures to collect metrics in the application internally. Alternatively, you can make only 1 worker per instance and use such an abstraction as orchestration (Kubernetes, opennebula, and others). And next, use alligator as a push server for metrics. Alligator has many collectors to collect it: pushgateway prom-aggregation-gateway statsd graphite file These approaches ensure that metrics will be collected from all workers of the application Applications have to send metrics to alligator. To remove influence on one metric by different workers, you can use two different approaches: mark metric with the worker ID from prometheus_client import Counter, push_to_gateway, CollectorRegistry
    registry = CollectorRegistry()
    c = Counter('my_requests_total', 'HTTP Failures', ['method', 'worker_id'], registry=registry)
    c.labels(method='get', worker_id=uwsgi.worker_id()).inc()
    c.labels(method='post', worker_id=uwsgi.worker_id()).inc()
    push_to_gateway('localhost:1111', job='myapp', registry=registry) use instead of metric type if you can. File collector cannot calculate increment, but others can increment if the counter is a type of metric. You can only send increment value to alligator, and alligator will calculate the sum. Counter Gauge More information about the difference between metric types can be found or on the . here official page What Metrics Do I Recommend to Measure in Applications? Firstly, it is any outgoing requests to external systems. In worker applications, it often is the reason to overflow workers or cause of low speed of query processing. My recommended list of metrics: count of requests to external services grouped by request types request time from external services (avg, or if possible, by quantiles/buckets of time) grouped by request types status of requests grouped by request type (success or errors) It would be nice if you could group not only by request type but more by network layer of requests: DNS resolve count, time, and status connect count, time, and status write/read count, time, and status It is also important to measure internal application metrics: ingoing requests grouped by types processing ingoing request time avg (quantile or buckets if possible) by type status of ingoing request by type cache status (hit/miss if it exists) I guess you can make another list of metrics, but these metrics are the most important to reveal problems and give main observability about application health. If your application is working in podman or docker environment, you can expose the container statistics. Alligator mechanics are similar to Cadvisor's when collecting container statistics. To enable, you have to use option cadvisor in the system context: Container checks. system {
    cadvisor;
} . This is a good way to monitor applications or external sources for reliability. Let’s check the resources on two ports every minute: Blackbox checks aggregate_period 60;
aggregate {
    blackbox http://127.0.0.1:80/check;
    blackbox https://127.0.0.1:443/check;
} Alligator produces network and application response metrics for this resource (response code, response time, and certificate expiration date). If the application has a page with JSON, alligator can also convert JSON to metrics: aggregate {
    jsonparse https://jsonplaceholder.typicode.com/todos;
} The next level of blackbox metrics - checking the process, service, and listening port: system {
	services uwsgi.service;
	process uwsgi;
}


query {
	expr 'count by (src_port, process) (socket_stat{process="uwsgi", src_port="80"})';
	make socket_match;
	datasource internal;
} And in the end: launching external programs and proactive actions (like watchdog): aggregate {
	process 'exec:///usr/local/bin/mycheckscript.sh';
} If the script can produce a metric that can determine the exact application downtime, this metric can be used for proactively restart: query {
    expr 'myapplication_status < 1';
    make app_is_down;
    datasource internal;
    action app_is_down;
}
action {
    name app_is_down;
    type shell;
    expr 'exec://systemctl restart uwsgi;
    datasource internal;
} System Metrics Any application metrics are also important. But without system metrics of servers, you won’t be able to clearly understand what happens in your system. System metrics are a set of metrics that help us gain insight into the health of our systems. This is the basis for making decisions if something goes wrong. An alligator with an empty configuration doesn’t collect any metrics at all. Any system metrics are classified into the following categories: base - CPU, memory, time, logical resources, and other common metrics about the server disk - disk usage and I/O network - interface statistics, network counters, sockets statistics smart - collecting S.M.A.R.T. statistics cpuavg - analog for load average when alligator make by CPU-only load firewall - counters from the system firewall process - check process (uptime, ulimits usage, status, CPU, memory, and disk consumption) packages - metrics with a list of installed packets in the system services - systemd services stats This is configured in a block called “system” in alligator.conf: system {
    base;
    disk;
    network;
    process;
    cpuavg period=5;
} What about cron jobs? How can it be connected to the Prometheus via alligator? There are several ways to do this. The simple way is to output metrics to a file: aggregate {
    prometheus_metrics file:///var/lib/stats/metrics.txt state=save;
} The parameter called ‘state’ enables autosaving at a point of the file when alligator is stopped. It makes us sure that we can stop the alligator and run again, and it rereads only that part of the file that has not been read yet. If we want to reread a file from the start (every 10 sec), we have to use . The third way - is to read only realtime data. To enable this mode, we have to remove any state from the configuration file. state=begin The other way is to use statsd or extended pushgateway protocol (called prom-aggregation-gateway): It only needs to have this section in the configuration: entrypoint {
	tcp 1111;
    metric_aggregation count;
} For instance, we have a code: #!/bin/sh
TASKS=0
do_something() {
        echo 0
        TASKS=$1
}

res=`do_something 1`
echo "amount_of_done_tasks {method=\"do_something\", status=\"done\"} $TASKS
task_exit_code {method=\"do_something\"} $res" | curl --data-binary @- http://localhost:1111/ This script will make two metrics in alligator: amount_of_done_tasks {status="done", method="do_something"} 0.000000
task_exit_code {method="do_something"} 0.000000 Sometimes, we might get a situation when the script is not working, but the metric exists (because alligator doesn’t know that the application doesn't work). To solve this problem, alligator has TTL mechanics. You can specify the lifetime in seconds for a request: curl -H "X-Expire-Time: 30" --data "living_thirty_seconds_metric 4" http://localhost:1111/ Or in the configuration file: entrypoint {
    ttl 30;
    tcp 1111;
    metric_aggregation count;
} The next way is using statsd protocol (udp or tcp). TCP server for statsd is enabled by default. If you want to use udp, specify the UDP port in config: entrypoint {
        ttl 30;
        tcp 1111;
        udp 1111;
} And your job script might have a section like this: from statsd import StatsClient, TCPStatsClient
from time import sleep

statsd = StatsClient(host='localhost',
                     port=1111,
                     prefix=None,
                     ipv6=False)

statsd.incr('baz')
statsd.incr('baz') Graphite protocol is also supported by alligator, and it works like graphite_exporter. Internet Web Server If I have an internet web server, what options do I have? First of all, https is the most popular protocol for web servers today. It forces us to check the certificate expiration time: To check directories with certificates: x509
{
	name nginx;
	path /etc/nginx/ssl/;
	match .crt;
}
x509
{
	name letsencrypt;
	path /etc/letsencrypt/live;
	match .pem;
} Secondly, to check the nginx listening ports: query {
	expr 'count by (src_port, process) (socket_stat{process="nginx", src_port="80"})';
	make socket_match;
	datasource internal;
}
query {
	expr 'count by (src_port, process) (socket_stat{process="nginx", src_port="443"})';
	make socket_match;
	datasource internal;
} To check nginx -t to exclude the situation, when nginx after server reboot cannot be started: aggregate {
	process 'exec:///sbin/nginx -t';
} To check system, web-server process, and firewall counters system {
    base;
    disk;
    network;
    cpuavg period=5;
    services nginx.service;
    process nginx;
    firewall ipset=on;
} We also make alligator port accessible only on local interfaces: entrypoint {
    handler prometheus;
    allow 127.0.0.1;
    allow 10.0.0.0/8;
    allow 172.16.0.0/12;
    allow 192.168.0.0/16;
    tcp 1111;
} or you can use the notation tcp in the tcp directive. <localIP>:<port> Statsd Metric Mapping Simple statsd doesn’t work with labels. To create a label from a statsd metric name, you must use mapping with metric name templates to change new names and create new labels. Due to this, I recommend using an extended format of Statsd, like DogStatsd (alligator also supports this extension), which supports labels in statsd protocol. But, some programs have already made pure statsd protocol without labels. One such example is Airflow. You can catch metrics from Airflow like this: airflow.ti.finish.upload_job.load_data.success:1|c
airflow.ti.finish.upload_job.other_data.success:1|c where the name is the following template: airflow.ti.finish.<job name>.<process name>:<increment by>|type The mapping for such metric name can be: entrypoint {
    tcp 1111;
    udp 1111;
    mapping {
        template   airflow.ti.finish.*.*.*;
        label dag_id "$1;
        label task_id "$2";
        label state "$3";
        name airflow_ti_finish;
        match glob;
    }
} This mapping will create such a metric: airflow_ti_finish {state="success", dag_id="upload_job", task_id="load_data"} 1.000000
airflow_ti_finish {state="success", dag_id="upload_job", task_id="other_data"} 1.000000 Statsd or Pushgateway Cluster Firstly, you can only make some amount of servers statsd or pushgateway and simply proxying to them. The first problem with using this: if you want to use gauges, you will have problems with calculating. If there is a distributed application sending a metric with the same name and labels, the gauge metric will also be replaced by the application. The first way to fix this - is to specify different labels for each application (such as hostname and/or worker_id/process_id). But that is not all. You also have a problem with the same gauge metric: this metric can be a balancer on different machines. Here is what to do with a metric that was sent at different times, but now they are interfering with each other: 10:12 http_request_time_avg {appid="43"} 12
10:14 http_request_time_avg {appid="43"} 22
10:55 http_request_time_avg {appid="43"} 122
11:58 http_request_time_avg {appid="43"} 1232 You can use nginx upstreams with a balancing method called hash: upstream alligators {
    server srv1.example.com:1112;
    server srv2.example.com:1112;
    server srv3.example.com:1112;
    server srv4.example.com:1112;
    server srv5.example.com:1112;
    hash $remote_addr consistent;
} Then one backend will always push all metrics to one server. On the other hand - cluster mode in alligator. Let's set it up! Firstly, we allocated 3 ports: for getting metrics 1111 for replication works 1112 for getting metrics from other applications 80 Imagine that our server is listening to the internet. Then, we need to secure our push metric ports. We can use the directive “return” with arg “empty” to remove any answers outside. We also know that internet browsers don't send POST queries to other domains if CORS doesn't allow this. Thirdly, make an authentication for POST requests. entrypoint {
   allow 127.0.0.1;
   allow 10.0.0.0/8;
   tcp 1111;
}
entrypoint {
   allow 10.0.0.0/8;
   tcp 1112;
   cluster replication;
   instance srv1.example.com:1112;
}
entrypoint {
   return empty;
   allow 0.0.0.0/0;
   ttl 300;
   tcp 80;
   metric_aggregation count;
   auth basic user:setme;
   auth_header Authorization;
   header access-control-allow-headers 'Authorization';
   header access-control-allow-methods POST;
   header access-control-allow-origin *;
} That's good. We annotate that we have a cluster called “replication” and the current instance name. Since this moment, alligator allows to answer collected oplog to anyone from network 10.0.0.0/8 on port 1112. How to command an alligator to join other instances of cluster? For this, we will use the “cluster” directive. We can specify oplog size (the limit of in-memory stored metrics that will wait to sync to other instances), and distribute metrics by its metric name (sharding key in our case, but you can use any other label). The replication factor hides replicated metrics while another primary server is running in the cluster. If the primary server goes down, hidden metrics will be exposed and made available to Prometheus from other instances. cluster {
   name replication;
   size 10000;
   sharding_key __name__;
   replica_factor 2;
   type oplog;
   servers srv1.example.com:1112 srv2.example.com:1112 srv3.example.com:1112 srv4.example.com:1112 srv5.example.com:1112;
} Check the External Sites Status Blackbox mechanics for collecting information about network services have already been mentioned. Nevertheless, that is not all that alligator can be used for. Alligator also has a puppeteer parser for loading external pages with a lot of site resources (which can load a page DOM). puppeteer {
    https://site1.example.com;
    https://site2.example.com;
    https://example.com;
} They might measure domain lookup name in metric: puppeteer_domainLookupEnd - puppeteer_domainLookupStart Connect time: puppeteer_connectEnd - puppeteer_connectStart Time to load DOM: puppeteer_DomContentLoaded Status of loading: puppeteer_eventSourceResponseStatus And even errors in the browser console: puppeteer_eventConsole This is not all that a puppeteer collector can do. However, this engine needs nodejs to be installed on a machine with puppeteer dependencies. Install puppeteer dependency: cd /var/lib/alligator/
npm i puppeteer@14.3.0
npm i ps-node-promise-es6 Conclusion Monitoring systems is a really big challenge despite the amount of tools to solve it. You can choose between a monitoring system or tools in one system of monitoring (such as VictoriaMetrics, Thanos, some amount of exporters, and so on). I guess that the approaches outlined in the article can be applicable to any other tools of monitoring. Alligator is a tool that can simplify it. The main goal of any monitoring is to make your system observable. You must have all measured information from your systems to ensure that it always operates correctly.

Alligator Is a Prometheus Monitoring Agent: Everything You Need to Know About It

“A few years back, I was tuning an alert rule and accidentally triggered the alert, which created 772 tickets. Twice.” This (all too true) story serves as the introduction of the main thesis of my talk in the video below (and the reason for its title): That alerts — contrary to the popular opinion held by IT practitioners across the spectrum of — don’t inherently suck. The problem lies in how alerts are typically created, which causes them to be… well, let’s just say “sub-optimal” and leave it at that. tech https://www.youtube.com/watch?v=K658JEZUGF4&embedable=true I’ve given this talk frequently at conferences such as , , , and others. I believe its popularity is largely due to its fun approach to a frustrating issue. DevOpsDays BRUM DevOpsDays TLV Monitorama I’d like to take a few moments of your time here to emphasize points I make in the talk but then extend those ideas in ways that don’t fit the limitations of time or format common in conference presentations. The slippery slope to “Monitoring Engineer” If you’ve read this far, there’s a good chance you care about alerts for more than just your own personal reasons. You probably have people — whether on your immediate team or in the larger organization — who look to and rely on you for help designing, implementing, maintaining, and fixing alerts. While most of us first encounter monitoring solutions because we want to know more about our own sh… tuff, it quickly follows that we’re helping others set up monitoring for themselves. Before long, we found ourselves in the “resident expert” role. Once that reputation gets around, the job (whether official or not) is irrevocably added to our responsibilities. The good news is that this is a huge opportunity for those who enjoy the work. Monitoring is an undeniable game-change in organizations willing to embrace and use it. Alerts ≠ Monitoring One of my first encounters with alerting that were completely off the rails was at a company that defined uptime as “100% minus the # of alerts” in a given period. It was utterly unhinged. While it was an extreme example, the underlying issue — confusing alerting with monitoring — isn’t rare at all. For many individuals (and teams, departments, and entire companies), the raison d’être for monitoring is to have alerts, which is simply not helpful or effective. Monitoring is nothing more (and nothing less) than the consistent, persistent collection of data from a set of systems. Everything else that a monitoring and observability solution provides — dashboards, widgets, reports, automation, and alerts — is merely a happy by-product of having monitoring in the first place. As a monitoring engineer, I know something is amiss when I see people hyper-focusing on alerts to the exclusion (if not the detriment) of monitoring. Alerts need proof of their value An alert should only exist if it has a proven, measurable, meaningful impact. The best way to validate that is to see if an alert is intended to cause someone to do something RIGHT. NOW. about a problem If all of those conditions aren’t met, you’re looking at an alert that is trying to replace some other monitoring structure — a dashboard, a report, etc. But that merely proves that an alert is actionable, not valuable. And I must be clear: “important” isn’t the same as “valuable.” Importance implies that it is technically, intellectually, or (believe it or not) emotionally meaningful to some person or group. “Valuable” is much more particular: The existence of the alert can be directly tied to a financial outcome. How does one establish this? Start with what the world would look like without the alert: How would the people who can fix the issue find out about the problem? And more to the point, how LONG would it take for the people who can resolve the issue to find out? Are there any inherent losses while the problem is happening? An online sales system that generates $1,000 an hour loses that amount every hour it’s unavailable. How long would it take to fix the problem? In some cases, it’s the same amount of time, alert or not. But in far more circumstances, if the problem were left unaddressed for the length of time identified in the first bullet, it would take longer (possibly significantly longer) to resolve. What is the regular (“total loaded”) rate for the staff who can fix the issue? What is the “interruption cost” for that staff? This means the staff is (ostensibly) not sitting around waiting for this particular error. So what is the value of their normal work? Because they will NOT be doing it while they are addressing this issue. You are welcome to take the formula above and, as the saying goes, “salt to taste.” Once you have this, recalculate all of the above WITH the alert. The difference between the first calculation and the second is the dollar value of the alert. Now, you can set up a simple report showing the number of occurrences the alert triggered multiplied by the value. That is the amount this one alert has saved the company during that time. Observability enables us to change our focus Back when I started working with monitoring solutions (yeah, yeah, Grampa. When dinosaurs ruled the earth and you had to chisel each bit into the hard drive by hand with a lodestone), we had to guess at the user’s experience from an array of much lower-level data. We’d look at network traffic, disk I/O, server connections, and other metrics and use those metrics to guess what was happening at the top of the OSI model. We didn’t do it because we thought it was the best option. We did it because it was the ONLY option. Tracing didn’t really come onto the scene — in terms of true application monitoring - until 2010. And it only took hold because of the fundamental change in application architecture. The widespread adoption of cloud computing (AWS EC2 went GA in 2006) and mobile phones (the first iPhone came on the scene in 2007) radically changed how we interacted with applications. Facebook had an unbelievable (for the time) 600 million users in 2010. That number grew to 800 million in 2011 and over 1 billion in 2012. Against THAT backdrop, application tracing and real user monitoring went from something we could only do in carefully controlled QA environments to a technique that was not only possible but game-changing. Because the entire reason we have monitoring — the whole damn point — is to understand what users are experiencing with an application. That’s it. That’s the whole enchilada. So, I will go on record as saying that alerting should focus on that aspect first and foremost. If the user experience is impacted, sound the alarm and get people out of bed if necessary. At that point, all the other telemetry - metrics, events, and logs - can be used to understand the details of the impact. But those lower-level data points no longer have to be the trigger point for alerts. Not in most cases. Where do we go from here? Hopefully, you have enough time between this blog and my talk to reflect on your existing alerts with an eye toward real improvement. You may find yourself deleting alerts you once thought essential. You will also undoubtedly spend time tweaking your alerts to make them more actionable, meaningful, and valuable. Just ensure you don’t trigger an alert storm in the process, or you’ll end up in the helpdesk, manually closing 1,544 tickets. Don’t ask me how I know. Also published . here

IT Monitoring Solutions: Your Alerts Shouldn't Be This Much of a Headache

Bridging the Test Coverage Gap With Proactive Monitoring in Production and Testing Environments

উত্পাদন এবং পরীক্ষার পরিবেশে সক্রিয় পর্যবেক্ষণের মাধ্যমে পরীক্ষার কভারেজ গ্যাপ পূরণ করা

Überbrückung der Testabdeckungslücke durch proaktive Überwachung in Produktions- und Testumgebungen

Cerrar la brecha en la cobertura de pruebas con monitoreo proactivo en entornos de producción y pruebas

Combler le déficit de couverture des tests grâce à une surveillance proactive dans les environnements de production et de test

उत्पादन और परीक्षण परिवेश में सक्रिय निगरानी के साथ परीक्षण कवरेज अंतर को पाटना

実稼働環境とテスト環境でのプロアクティブなモニタリングによりテスト カバレッジのギャップを埋める

프로덕션 및 테스트 환경에서 사전 모니터링을 통해 테스트 범위 격차 해소

Preenchendo a lacuna de cobertura de teste com monitoramento proativo em ambientes de produção e teste

Устранение разрыва в тестовом покрытии с помощью упреждающего мониторинга в производственных и тестовых средах

Üretim ve Test Ortamlarında Proaktif İzleme ile Test Kapsamı Açığının Kapatılması

Thu hẹp khoảng cách về phạm vi kiểm thử bằng tính năng giám sát chủ động trong môi trường sản xuất và thử nghiệm

通过生产和测试环境中的主动监控来缩小测试覆盖率差距

Intro Data Quality is a vast topic that consists of different parts, some of which include: Schema validation; Data cleaning; Data profiling; Unit tests for Data; Monitoring; and others. In this article, I will explain how Amazon Deequ could be used for monitoring data quality (based on regular profiling). For example, we will: Take a CSV file with data; Make profiling rules with Deequ; Write results to InfluxDB; Visualise results in Grafana. Introduction to Amazon Deequ & Metric Repository According to Amazon Deequ developers, Deequ is a library built on top of Apache Spark for defining "unit tests for data.” It measures data quality in large datasets. The entire project is located . here We can use Deequ to calculate different metrics (such as CountDistinct, Distinctness, Maximum, Mean, Min, Uniqueness, and others) against various datasets. All calculations work on top of Apache Spark, making the library effective and scalable. The Metric Repository is an interface that allows the saving of computation results to different systems/formats. The library offers the capability to save results as a DataFrame or to a file system as JSON or in memory. You can extend this interface and add implementation for different systems/formats. In this article, we will save our results to InfluxDB for further visualization in Grafana. Environment To run the example that we will create in the article, you need the following: Apache Spark versions 2.2.x to 2.4.x and 3.0.x. Docker Scala/Java Let’s run InfluxDB (run the next commands in the console): # pull influxDB docker image
docker pull influxdb:1.8.4
# run docker container
docker run — rm — name=influxdb -d -p 8086:8086 influxdb:1.8.4
# connect to docker container and run influxDB console
docker exec -it influxdb influx
# create database which we will use for storing our metrics
create database example Let’s run Grafana (use new console tab): # pull grafana docker image
docker pull grafana/grafana
# run Grafana docker container & connect it with influxDB container
docker run -d — rm — name=grafana -p 3000:3000 — link influxdb grafana/grafana Now, you can clone this from GitHub and open it in your favorite IDE. repository Under path: you can find MetricRepository for InfluxDB — we will use it for our task. src/main/scala/lexaneon/amazon/deequ/repository/influxdb , Under example folder: you can find the full example, which is discussed below src/main/scala/lexaneon/amazon/deequ/example/InfluxDBMetricRepository.scala package lexaneon.amazon.deequ.example

import com.amazon.deequ.analyzers.{Completeness, CountDistinct, Distinctness, Size, Uniqueness}
import com.amazon.deequ.analyzers.runners.AnalysisRunner
import com.amazon.deequ.analyzers.runners.AnalyzerContext.successMetricsAsDataFrame
import com.amazon.deequ.repository.ResultKey
import lexaneon.amazon.deequ.repository.influxdb.{InfluxDBConnectionProperties, InfluxDBMetricsRepository}
import org.apache.spark.sql.SparkSession

object InfluxDBMetricRepository extends App{

  val spark = initSpark()
  // here is our data file, which we use for this example, you can get your own file.
  val filePath = "src/main/resources/dataForExample/data.csv" 

  val df = spark.read.option("header", "true").csv(filePath).toDF()

  /**
  * "http://localhost:8086" - URL to InfluxDB, if you do not change - it should be the same
  * "example" - database name, which was created before
  * "InfluxDBMetricsRepository" - table (measurement) name which is used for data
  */
  val influxDBConnectionProperties = InfluxDBConnectionProperties("http://localhost:8086", "example", "InfluxDBMetricsRepository")

  /**
  * Object resultKey is information that uniquely identifies a AnalysisResult
  * @param dataSetDate A date related to the AnalysisResult
  * @param tags        A map with additional annotations, you can add your own tags, 
  * they will be stored in column (tag) with name: tags_
  */
  val resultKey = ResultKey(
    System.currentTimeMillis(),
    Map("dataSetFilePath" -> filePath, "dataSetName" -> "orders"))

  val analysisResult = AnalysisRunner
    .onData(df) // pass our DataFrame
    .useRepository(new InfluxDBMetricsRepository(influxDBConnectionProperties))
    .saveOrAppendResult(resultKey) 
    .addAnalyzer(Size()) // here we setup which metrics should be calculated
    .addAnalyzer(Distinctness("customer_id"))
    .addAnalyzer(CountDistinct("customer_id"))
    .addAnalyzer(Uniqueness(Seq("customer_id", "id")))
    .addAnalyzer(Uniqueness("id"))
    .addAnalyzer(Completeness("trans_date"))
    .addAnalyzer(Completeness("id"))
    .run()

  val metric = successMetricsAsDataFrame( spark, analysisResult)

  metric.show(false) // output to console

  spark.close()
  def initSpark(isLocalRun: Boolean = true): SparkSession = {
    val sparkSessionBuilder =
      SparkSession
        .builder
        .appName(this.getClass.getSimpleName)

    val spark =
      if (isLocalRun){
        sparkSessionBuilder
          .master("local[*]")
          .getOrCreate()
      }else
        sparkSessionBuilder.getOrCreate()

    spark.sparkContext.setLogLevel("ERROR")
    spark.conf.set("spark.sql.session.timeZone", "UTC")

    spark
  }

} when we execute InfluxDBMetricRepository object — we can go to the console where we connected to InfluxDB and check the results: use example
select * from InfluxDBMetricsRepository We have to see the same results: If you run this example several times agains same data, then you will find results for each execution, btw at InfluxDB you can define retention period (how much time data should be stored). At the IDE console output you find out something like this: Now it is time to create a dashboard in Grafana. Open the link at your browser: (default user/password admin/admin) http://localhost:3000/ let’s create a new Data Source for our InfluxDB. Go to configuration — > Data Sources — > Add data source Name: InfluxDB
URL: http://influxdb:8086
Database: example
User: admin
Password: admin Press “Save & Test” Then, define query parameters, You can also play with other dashboard parameters to achieve the necessary results. Conclusion You can use Amazon Deequ not only as the library for Unit Tests but also for creating a regular Data quality monitoring process. For example, I’ve created an adapter for InfluxDB. However, you can create your own or use either my InfluxDB MetricRepository or FileSystemMetricRepository from Amazon (then process JSON files with some tool that you prefer more). Of course, you can build derived metrics and use them to visualize or even send notifications to data Stewart. References Amazon Deequ project page Test data quality at scale with Deequ — link InfluxDB MetricRepository & example — link InfluxDB key concepts — link Also published . here

A Guide For Data Quality Monitoring with Amazon Deequ

MONEYTOR

moneytor.in

-- employees

Since n.d.

COMPANY RANKING

MONEYTOR

EVERGREEN INDEX #6612

Representing this company? Click here to claim and customize this page!

Moneytor WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

MONEYTOR

moneytor.in

-- employees

Since n.d.

COMPANY RANKING

MONEYTOR

EVERGREEN INDEX #6612

RELATED COMPANIESRANK

STORIES ABOUT MONEYTOR

Representing this company? Click here to claim and customize this page!

Moneytor WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!