LoanShout

loanshout

If you’ve migrated from a monolith to a microservices architecture you've probably experienced this sentiment: Microservices combined with containerized deployment results in highly dynamic systems with many moving parts across multiple layers. Modern systems today are far more complex to monitor. These systems emit massive amounts of highly dimensional telemetry data from hardware and the operating system, through Docker and Kubernetes, all the way to application and business performance metrics. Many have come to realize that the commonly prescribed Graphite+StatsD monitoring stack is no longer sufficient to cover their backs. Why is that? And what is the current approach to monitoring? In this post, I will look at the characteristics of modern systems and the new challenges they raise for monitoring. I will also discuss common solutions for monitoring, from the days of Graphite and StatsD to the currently dominant Prometheus. I’ll review the benefits of Prometheus as well as its limitations, and how open source time-series databases can help overcome some of these limitations. The challenges of monitoring modern systems The Challenges of Monitoring Modern Systems Let’s have a look at the changes in software systems and the challenges they introduce in system monitoring. 1. Open source and cloud services Open source and SaaS increased the selection of readily available libraries, tools, and frameworks for building software systems, including web servers, databases, message brokers, and queues. That turned system architecture into a composite of multiple third-party frameworks. Moreover, many frameworks are now clustered solutions with their own suite of moving parts (just think Hadoop or Kafka, compared to traditional MySQL). Monitoring systems now need to provide integrations with a large and dynamic ecosystem of third-party platforms to provide complete observability. The Challenge: 2. Microservices instead of a monolith Following a microservice architecture, a typical monolith application would be broken down into a dozen or more microservices, each one potentially running its own programming language and database, each one independently deployed, scaled, and upgraded. Uber for example over 4,000 proprietary microservices and a growing number of open source systems which posed a challenge for their monitoring system. reported in late 2014 : A surge in the number of discrete components you need to monitor. The Challenge The complexity of microservices illustrated through the Amazon and Netflix Deathstars (credit: Amazon and Netflix ) 3. Cloud-native architecture and Kubernetes Cloud-native architectures, based on containers and Kubernetes, have grown in popularity for running microservices, but have also added yet another layer of complexity. Now you need to monitor applications spanning multiple containers, pods, and namespaces, potentially over fleets of clusters. The containers framework itself is now a vital system you need to : Kubernetes cluster metrics, node metrics, pod, and container metrics, as well as Kubernetes’ own control plane services. monitor : In containerized workloads, you have to monitor multiple layers and dimensions: The Challenge Infrastructure metrics, such as host CPU and memory Container runtime and Kubernetes metrics, such as running pods and node resource utilization Application metrics, such as Request Rate and Duration Background on monitoring tools Background on Monitoring Tools Now that we saw WHAT we need to monitor within our systems, let’s talk about HOW we can monitor them. Not so long ago a common monitoring practice in the industry was based on the open source combination of , . In this setup system components send metrics to StatsD in a fairly simple way. StatsD performs some aggregation and then flushes them to Graphite. Graphite uses Carbon to receive and aggregate the metrics from StatsD, then stores them in its Whisper time-series database. Grafana provides dashboarding and visualization on the data. Graphite StatsD and Grafana Modern systems brought new requirements which existing solutions weren’t equipped to handle. In fact, when the engineering team at SoundCloud moved to a microservices architecture and encountered with their existing Graphite and StatsD monitoring, that drove them to create the Prometheus open source project. limitations Since then, Prometheus and Grafana open source have emerged as a . Prometheus is ranked 66th place on , while Graphite is behind in 78th place, per the November 2020 report. Currently, ranks 32k stars on GitHub with 4.9k forks, while ranks 36.1k stars with 7.2k forks. popular combination for monitoring DB-Engine Prometheus Grafana In a way, the transition from Graphite to Prometheus represents the transition to modern monitoring practices. The Solution: New Monitoring Capabilities For Microservices As we saw, systems have undergone a significant transformation that challenged traditional monitoring approaches. In order to meet these challenges, modern monitoring systems had to gear up with new capabilities: Flexible querying with high cardinality Efficient automated metric scraping Scalability to handle a large volume of metrics Let’s look into these new requirements and the available monitoring approaches to meet them: Flexible querying over high cardinality The Solution: The introduction of flexible querying based on tags makes it a lot easier to slice and dice the data along multiple dimensions like service, instance, endpoint, and method. Back in the Graphite days, we used to work with a hierarchical dot-concatenated metrics naming format. For example, a Kafka metric may look something like this: PROD.KafkaBrokers.us-east-1.1a.kafka1.load.load.shortterm The hierarchical naming worked well for static machine-centric metrics, but it turned out to be too strict and cumbersome to express the dynamic services metrics with the high cardinality of modern systems. For example, cloud-native environments assume that at any given moment a container may crash, a pod may be evicted, or a node may die, resulting in a hostname or instance name change, which in Graphite will result in a new set of metrics. And there’s also the dynamics of deployment changes. For example, If system dynamics require another level in the metrics hierarchy, such as when moving from a single region to multi-region (like “us-east-1” in the middle of the metric name in the above example), it will “break” existing queries, aliases and regex, which are based on the old hierarchy. Querying across hierarchies, such as error rates across all services, can result in queries which are hard to compose and later hard to understand and maintain. Prometheus offered a tagged data model from the get-go in 2015, which swept the market. Graphite reacted with delay and launched its own tags support in 2018. Metric scraping and auto-discovery The Solution: automatic service discovery and scraping metrics off of the services saves a lot of hassle in getting your metrics into your monitoring system. The common practice by StatsD and other traditional solutions was to collect metrics in push mode, which required installing local daemons and explicitly configuring each component and 3rd party tool with the metrics collector destination. With the many frameworks and languages involved in modern systems, however, it has become challenging to maintain this explicit push-mode sending of metrics. Adding Kubernetes to the mix increased the complexity even further. Teams were looking to offload this work. This was a distinct strongpoint of Prometheus, which offered a pull-mode scraping, together with service discovery of the components (“targets” in Prometheus terms). In particular, Prometheus shined with its native scraping from Kubernetes, which skyrocketed in demand. As the popularity of Prometheus grew, many open source projects added support for Prometheus Metrics Exporter format, which has made metrics scraping with Prometheus even more seamless. Today you can find Prometheus for many common systems including popular databases, messaging systems, web servers, hardware components and operating systems. exporters Clustered scalable solution The Solution: current metrics volumes require a horizontally scalable storage that enables both long retention of historical data and high availability. As the volume of telemetry data started rising, high scale companies such as , Wayfair and Uber started experiencing scaling challenges with their existing Graphite and StatsD monitoring solution. As the Wayfair team : Taboola put it “Graphite/Carbon does not support true clustering (replication and high availability)”. Uber’s testimonial that: states “By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.” Prometheus did not manage to solve the scaling problem either. Prometheus comes as a single node deployment, storing the data on local storage. This limits the total amount of data you can store, which translates to limits on the historical data you can query and compare against. Uber engineering, for example, had several teams that ran significant Prometheus deployments, but then reported: “As our dimensionality and usage of metrics increases, common solutions like Prometheus and Graphite become difficult to manage and sometimes cease to work.” The need for historical data in Prometheus often forces aggressive downsampling strategies, which may severely limit the ability to identify and analyze outliers in the historical data. Alternatively, organizations find themselves manually sharding their metrics across several discrete Prometheus instances, having to deal with challenges of distributed systems. Time Series Databases (TSDB) to the rescue The sheer volume of telemetry data in modern systems calls for new scalable solutions. Prometheus doesn’t meet the scaling challenge. Modern time-series databases (TSDB) offer a variety of robust scalable solutions to address this challenge. The popularity of TSDB has skyrocketed in the past couple of years, any other type of database. A variety of use cases have helped with this uptick in popularity such as systems metrics collection, IoT, and financial services. Alongside general-purpose TSDB, some are more specialized, including those designed primarily for the purposes of metrics monitoring. far beyond Database trends from January 2019 through October 2020 showing the large increase in demand for time-series databases (credit: DB-engines ) Many new s have emerged in the past few years, a significant percentage of which are open-source. In fact, open-source appears to be a main driving force behind TSDB, with some 80% of TSDB in use are open-source based, OSS rate among all database types. TSDB the highest Comparison of database popularity by open-source versus commercial licenses (credit: DB-engines ) TSDB as Prometheus Long Term Storage (LTS) The new generation of TSDB can serve as long-term storage (LTS) solutions for Prometheus, so you can enjoy the benefits of Prometheus’ metrics scraping and query capabilities while supporting long-term data retention and the wealth of Grafana dashboards built for the Prometheus data source. The standard way in which TSDB integrates with Prometheus as an LTS is via Prometheus’ feature. Prometheus’ remote write allows sending samples to LTS and takes care of relabeling, buffering (it can run multiple queues for multiple remote endpoints), authorization, and other aspects. remote write When choosing a TSDB for a Prometheus LTS, we look for: Prometheus compatibility, supporting (Prometheus Query Language) and the respective APIs PromQL Horizontal scalability and high availability to handle large-scale metrics Downsampling of data with regular roll-ups and aggregation Backfilling and out of order writes (for network lags or retrospective historical feed) Multi-tenancy to support multiple teams, business units, customers, partners etc. Many modern open source TSDB offer integration with Prometheus. Within the Cloud Native Computing Foundation (CNCF) incubation projects, you can find both and , with some cross-pollination between the two. Other open source TSDB include by Uber, and by InfluxData, to name a few. These solutions take different approaches and present different tradeoffs as Prometheus LTS. Some of the solutions come with their own UI, data collection mechanisms, and even query language variants, however using Prometheus and Grafana remains the common practice. Cortex Thanos M3DB VictoriaMetrics InfluxDB Some of these TSDBs are fairly young OSS projects, with all the implications on maturity, documentation, and accumulated expertise in the user community. If you require Prometheus long term storage for your production system, TSDB is a good path, though I highly recommend making a thorough evaluation based on your workload, needs, and tolerance. Endnote Modern systems are highly dynamic in nature, and emit huge amounts of high cardinality telemetry data, especially when based on cloud-native and microservices architectures. These systems also make ever-increasing use of third-party open source tools and cloud services which need to be monitored as well. Prometheus sets a good standard with its metrics scraping and flexible tagged query language. This approach, best exemplified with the seamless integration with Kubernetes, encouraged many projects and tools to start offering Prometheus exporters and Grafana dashboards as a standard for their monitoring. Running only on a single node proved to be a severe limitation of Prometheus. Current monitoring needs a clustered solution that can hold historical data long term, without sacrificing data granularity with aggressive downsampling. Whether you’re choosing the monitoring system for a new project, or whether you’re facing scaling issues with your current monitoring solution, integrating TSDB as long-term storage for Prometheus enables you to enjoy the best of both worlds: metric scraping, querying and visualization based on Prometheus and Grafana while handling high scale of telemetry data. This article was originally published on InfoQ at 12/3/2020

Modern Day Challenges to Monitoring Microservices

In case you missed it, I using a from . Then I built a to serve the model’s predictions. That’s nice and all, but… how is my model? built a neural network to predict loan risk public dataset LendingClub public API good Today I’m going to put it to the test, pitting it against the risk models of the very institution who issued those loans. That’s right, LendingClub included their own calculated loan grades (and sub-grades) in the dataset, so all the pieces are in place for the most thrilling risk modeling smackdown of this century (or at least this week). May the best algorithm win! joblib

prev_notebook_folder = loans = joblib.load(prev_notebook_folder + )
loans.shape import "../input/building-a-neural-network-to-predict-loan-risk/" "loans_for_eval.joblib" (1110171, 70) loans.head() ┌────┬────────────┬────────────┬─────────────┬─────────────────┬─────────────┬─────────────────────┬────────┬──────────────┬───────────────────┬─────────────────┬──────┬────────────┬──────────────────┬────────────────────┬─────────────────┬─────────────────────────────┬─────────────────────┬───────────┬────────┬────────────┬─────────────────┐
│    │ loan_amnt  │   term     │ emp_length  │ home_ownership  │ annual_inc  │      purpose        │  dti   │ delinq_2yrs  │ cr_hist_age_mths  │ fico_range_low  │ ...  │ tax_liens  │ tot_hi_cred_lim  │ total_bal_ex_mort  │ total_bc_limit  │ total_il_high_credit_limit  │ fraction_recovered  │ issue_d   │ grade  │ sub_grade  │ expected_return │
├────┼────────────┼────────────┼─────────────┼─────────────────┼─────────────┼─────────────────────┼────────┼──────────────┼───────────────────┼─────────────────┼──────┼────────────┼──────────────────┼────────────────────┼─────────────────┼─────────────────────────────┼─────────────────────┼───────────┼────────┼────────────┼─────────────────┤
│ 0  │    3600.0  │ 36 months  │ 10+ years   │ MORTGAGE        │    55000.0  │ debt_consolidation  │  5.91  │         0.0  │            148.0  │          675.0  │ ...  │       0.0  │        178050.0  │            7746.0  │         2400.0  │                    13734.0  │                1.0  │ Dec-2015  │ C      │ C4         │         4429.08 │
│ 1  │   24700.0  │ 36 months  │ 10+ years   │ MORTGAGE        │    65000.0  │ small_business      │ 16.06  │         1.0  │            192.0  │          715.0  │ ...  │       0.0  │        314017.0  │           39475.0  │        79300.0  │                    24667.0  │                1.0  │ Dec-2015  │ C      │ C1         │        29530.08 │
│ 2  │   20000.0  │ 60 months  │ 10+ years   │ MORTGAGE        │    63000.0  │ home_improvement    │ 10.78  │         0.0  │            184.0  │          695.0  │ ...  │       0.0  │        218418.0  │           18696.0  │         6200.0  │                    14877.0  │                1.0  │ Dec-2015  │ B      │ B4         │        25959.60 │
│ 4  │   10400.0  │ 60 months  │ 3 years     │ MORTGAGE        │   104433.0  │ major_purchase      │ 25.37  │         1.0  │            210.0  │          695.0  │ ...  │       0.0  │        439570.0  │           95768.0  │        20300.0  │                    88097.0  │                1.0  │ Dec-2015  │ F      │ F1         │        17394.60 │
│ 5  │   11950.0  │ 36 months  │ 4 years     │ RENT            │    34000.0  │ debt_consolidation  │ 10.20  │         0.0  │            338.0  │          690.0  │ ...  │       0.0  │         16900.0  │           12798.0  │         9400.0  │                     4000.0  │                1.0  │ Dec-2015  │ C      │ C3         │        14586.48 │
└────┴────────────┴────────────┴─────────────┴─────────────────┴─────────────┴─────────────────────┴────────┴──────────────┴───────────────────┴─────────────────┴──────┴────────────┴──────────────────┴────────────────────┴─────────────────┴─────────────────────────────┴─────────────────────┴───────────┴────────┴────────────┴─────────────────┘
5 rows × 70 columns This post was adapted from a Jupyter Notebook, by the way, so if you’d like to follow along in your own notebook, go ahead and fork mine or ! Kaggle GitHub Ground rules This is going to be a clean fight—my model won’t use any data LendingClub wouldn’t have access to at the point they calculate a loan’s grade (including the grade itself). I’m going to sort the dataset chronologically (using the column, the month and year the loan was issued) and split it into two parts. The first 80% I’ll use for training my competition model, and I’ll compare performance on the last 20%. issue_d sklearn.model_selection train_test_split

loans[ ] = loans[ ].astype( )
loans.sort_values( , axis= , inplace= , kind= )

train, test = train_test_split(loans, test_size= , shuffle= )
train, test = train.copy(), test.copy()
print( ) from import "date" "issue_d" "datetime64[ns]" "date" "index" True "mergesort" 0.2 False f"The test set contains loans." {len(test):,} The test set contains 222,035 loans. At the earlier end of the test set my model may have a slight informational advantage, having been trained on a few loans that may not have closed yet at the point LendingClub was grading those ones. On the other hand, LendingClub may have a slight informational advantage on the later end of the test set, since they would have known the outcomes of some loans on the earlier end of the test set by that time. I have to give credit to Michael Wurm, by the way, for of comparing my model’s performance to LendingClub’s loan grades, but my approach is pretty different. I’m not trying to simulate the performance of an investment portfolio; I’m just evaluating how well my predictions of simple risk compare. the idea Test metric The test: who can pick the best set of grade A loans, judged on the basis of the independent variable from , the fraction of an expected loan return that a prospective borrower will pay back (which I engineered as ). my last notebook fraction_recovered LendingClub will take the plate first. I’ll gather all their grade A loans from the test set, count them, and calculate their average . That average will be the metric my model has to beat. fraction_recovered Then I’ll train my model on the training set using the same pipeline and parameters I settled on in . Once it’s trained, I’ll use it to make predictions on the test set, then gather the number of top predictions equal to the number of LendingClub’s grade A loans. Finally, I’ll calculate the same average of on that subset, and we’ll have ourselves a winner! my last notebook fraction_recovered LendingClub's turn statistics mean

lc_grade_a = test[test[ ] == ]
print( )

print( )
print(round(mean(lc_grade_a[ ]), )) from import "grade" "A" f"LendingClub gave loans in the test set an A grade." {len(lc_grade_a):,} "\nAverage `fraction_recovered` on LendingClub's grade A loans:" "fraction_recovered" 5 LendingClub gave 38,779 loans in the test set an A grade.

Average `fraction_recovered` on LendingClub's grade A loans:
0.96021 That’s a pretty high percentage. I’m a bit nervous. My turn First, I’ll copy over my function from : run_pipeline my previous notebook sklearn.model_selection train_test_split sklearn_pandas DataFrameMapper sklearn.preprocessing OneHotEncoder, OrdinalEncoder, StandardScaler tensorflow.keras Sequential, Input tensorflow.keras.layers Dense, Dropout X = data.drop(columns=[ ])
    y = data[ ]
    X_train, X_valid, y_train, y_valid = (
        train_test_split(X, y, test_size= , random_state= ) validate (X, , y, )
    )

    transformer = DataFrameMapper(
        [
            (onehot_cols, OneHotEncoder(drop= )),
            (
                list(ordinal_cols.keys()),
                OrdinalEncoder(categories=list(ordinal_cols.values())),
            ),
        ],
        default=StandardScaler(),
    )

    X_train = transformer.fit_transform(X_train)
    X_valid = transformer.transform(X_valid) validate input_nodes = X_train.shape[ ]
    output_nodes = model = Sequential()
    model.add(Input((input_nodes,)))
    model.add(Dense( , activation= ))
    model.add(Dropout( , seed= ))
    model.add(Dense( , activation= ))
    model.add(Dropout( , seed= ))
    model.add(Dense( , activation= ))
    model.add(Dropout( , seed= ))
    model.add(Dense(output_nodes))
    model.compile(optimizer= , loss= )

    history = model.fit(
        X_train,
        y_train,
        batch_size=batch_size,
        epochs= ,
        validation_data=(X_valid, y_valid) validate ,
        verbose= ,
    ) history.history, model, transformer


onehot_cols = [ , , , ]
ordinal_cols = { : [ , , , , , , , , , , ,
    ]
} from import from import from import from import from import : def run_pipeline (
    data,
    onehot_cols,
    ordinal_cols,
    batch_size,
    validate=True,
) "fraction_recovered" "fraction_recovered" 0.2 0 if else None None "if_binary" if else None 1 1 64 "relu" 0.3 0 32 "relu" 0.3 1 16 "relu" 0.3 2 "adam" "mean_squared_logarithmic_error" 100 if else None 2 return "term" "application_type" "home_ownership" "purpose" "emp_length" "< 1 year" "1 year" "2 years" "3 years" "4 years" "5 years" "6 years" "7 years" "8 years" "9 years" "10+ years" Now for the moment of truth: _, model, transformer = run_pipeline(
    train.drop(columns=[ , , , , ]),
    onehot_cols,
    ordinal_cols,
    batch_size= ,
    validate= ,
) X_test = transformer.transform(
    test.drop(
        columns=[ , , , , , ,
        ]
    )
)
test[ ] = model.predict(X_test) test_sorted = test.sort_values( , axis= , ascending= )
ty_grade_a = test_sorted.iloc[ :len(lc_grade_a)] print( )
print(format(mean(ty_grade_a[ ]), )) # Train the model "issue_d" "date" "grade" "sub_grade" "expected_return" 128 False # Make predictions "fraction_recovered" "issue_d" "date" "grade" "sub_grade" "expected_return" "model_predictions" # Gather top predictions "model_predictions" "index" False 0 # Display results "\nAverage `fraction_recovered` on Ty's grade A loans:" "fraction_recovered" ".5f" Epoch 1/100
6939/6939 - 13s - loss: 0.0249
Epoch 2/100
6939/6939 - 13s - loss: 0.0204
Epoch 3/100
6939/6939 - 13s - loss: 0.0202
Epoch 4/100
6939/6939 - 13s - loss: 0.0202
Epoch 5/100
6939/6939 - 13s - loss: 0.0202
Epoch 6/100
6939/6939 - 14s - loss: 0.0201
Epoch 7/100
6939/6939 - 14s - loss: 0.0201
Epoch 8/100
6939/6939 - 14s - loss: 0.0201
Epoch 9/100
6939/6939 - 13s - loss: 0.0201
Epoch 10/100
6939/6939 - 12s - loss: 0.0201
Epoch 11/100
6939/6939 - 13s - loss: 0.0201
Epoch 12/100
6939/6939 - 13s - loss: 0.0201
Epoch 13/100
6939/6939 - 13s - loss: 0.0201
Epoch 14/100
6939/6939 - 13s - loss: 0.0201
Epoch 15/100
6939/6939 - 12s - loss: 0.0201
Epoch 16/100
6939/6939 - 12s - loss: 0.0201
Epoch 17/100
6939/6939 - 13s - loss: 0.0200
Epoch 18/100
6939/6939 - 13s - loss: 0.0200
Epoch 19/100
6939/6939 - 13s - loss: 0.0200
Epoch 20/100
6939/6939 - 14s - loss: 0.0200
Epoch 21/100
6939/6939 - 13s - loss: 0.0200
Epoch 22/100
6939/6939 - 13s - loss: 0.0200
Epoch 23/100
6939/6939 - 12s - loss: 0.0200
Epoch 24/100
6939/6939 - 12s - loss: 0.0200
Epoch 25/100
6939/6939 - 12s - loss: 0.0200
Epoch 26/100
6939/6939 - 13s - loss: 0.0200
Epoch 27/100
6939/6939 - 13s - loss: 0.0200
Epoch 28/100
6939/6939 - 13s - loss: 0.0200
Epoch 29/100
6939/6939 - 13s - loss: 0.0200
Epoch 30/100
6939/6939 - 13s - loss: 0.0200
Epoch 31/100
6939/6939 - 15s - loss: 0.0200
Epoch 32/100
6939/6939 - 13s - loss: 0.0200
Epoch 33/100
6939/6939 - 12s - loss: 0.0200
Epoch 34/100
6939/6939 - 13s - loss: 0.0200
Epoch 35/100
6939/6939 - 13s - loss: 0.0200
Epoch 36/100
6939/6939 - 13s - loss: 0.0200
Epoch 37/100
6939/6939 - 13s - loss: 0.0200
Epoch 38/100
6939/6939 - 13s - loss: 0.0200
Epoch 39/100
6939/6939 - 13s - loss: 0.0200
Epoch 40/100
6939/6939 - 13s - loss: 0.0200
Epoch 41/100
6939/6939 - 13s - loss: 0.0200
Epoch 42/100
6939/6939 - 13s - loss: 0.0200
Epoch 43/100
6939/6939 - 14s - loss: 0.0200
Epoch 44/100
6939/6939 - 13s - loss: 0.0200
Epoch 45/100
6939/6939 - 13s - loss: 0.0200
Epoch 46/100
6939/6939 - 13s - loss: 0.0200
Epoch 47/100
6939/6939 - 13s - loss: 0.0200
Epoch 48/100
6939/6939 - 13s - loss: 0.0200
Epoch 49/100
6939/6939 - 13s - loss: 0.0200
Epoch 50/100
6939/6939 - 13s - loss: 0.0200
Epoch 51/100
6939/6939 - 13s - loss: 0.0200
Epoch 52/100
6939/6939 - 13s - loss: 0.0200
Epoch 53/100
6939/6939 - 13s - loss: 0.0200
Epoch 54/100
6939/6939 - 14s - loss: 0.0200
Epoch 55/100
6939/6939 - 14s - loss: 0.0200
Epoch 56/100
6939/6939 - 13s - loss: 0.0200
Epoch 57/100
6939/6939 - 13s - loss: 0.0200
Epoch 58/100
6939/6939 - 13s - loss: 0.0200
Epoch 59/100
6939/6939 - 13s - loss: 0.0200
Epoch 60/100
6939/6939 - 13s - loss: 0.0200
Epoch 61/100
6939/6939 - 13s - loss: 0.0200
Epoch 62/100
6939/6939 - 13s - loss: 0.0200
Epoch 63/100
6939/6939 - 13s - loss: 0.0200
Epoch 64/100
6939/6939 - 13s - loss: 0.0200
Epoch 65/100
6939/6939 - 12s - loss: 0.0200
Epoch 66/100
6939/6939 - 13s - loss: 0.0200
Epoch 67/100
6939/6939 - 14s - loss: 0.0200
Epoch 68/100
6939/6939 - 13s - loss: 0.0200
Epoch 69/100
6939/6939 - 13s - loss: 0.0200
Epoch 70/100
6939/6939 - 13s - loss: 0.0200
Epoch 71/100
6939/6939 - 13s - loss: 0.0200
Epoch 72/100
6939/6939 - 13s - loss: 0.0200
Epoch 73/100
6939/6939 - 13s - loss: 0.0200
Epoch 74/100
6939/6939 - 13s - loss: 0.0200
Epoch 75/100
6939/6939 - 13s - loss: 0.0200
Epoch 76/100
6939/6939 - 13s - loss: 0.0200
Epoch 77/100
6939/6939 - 13s - loss: 0.0200
Epoch 78/100
6939/6939 - 13s - loss: 0.0200
Epoch 79/100
6939/6939 - 14s - loss: 0.0200
Epoch 80/100
6939/6939 - 13s - loss: 0.0200
Epoch 81/100
6939/6939 - 13s - loss: 0.0200
Epoch 82/100
6939/6939 - 13s - loss: 0.0200
Epoch 83/100
6939/6939 - 13s - loss: 0.0200
Epoch 84/100
6939/6939 - 12s - loss: 0.0200
Epoch 85/100
6939/6939 - 13s - loss: 0.0200
Epoch 86/100
6939/6939 - 13s - loss: 0.0200
Epoch 87/100
6939/6939 - 13s - loss: 0.0200
Epoch 88/100
6939/6939 - 13s - loss: 0.0200
Epoch 89/100
6939/6939 - 13s - loss: 0.0200
Epoch 90/100
6939/6939 - 13s - loss: 0.0200
Epoch 91/100
6939/6939 - 14s - loss: 0.0200
Epoch 92/100
6939/6939 - 13s - loss: 0.0200
Epoch 93/100
6939/6939 - 13s - loss: 0.0200
Epoch 94/100
6939/6939 - 13s - loss: 0.0200
Epoch 95/100
6939/6939 - 13s - loss: 0.0200
Epoch 96/100
6939/6939 - 13s - loss: 0.0200
Epoch 97/100
6939/6939 - 13s - loss: 0.0200
Epoch 98/100
6939/6939 - 13s - loss: 0.0200
Epoch 99/100
6939/6939 - 13s - loss: 0.0200
Epoch 100/100
6939/6939 - 13s - loss: 0.0200

Average `fraction_recovered` on Ty's grade A loans:
0.96166 Victory! Phew, that was a close one! My win might be too small to be statistically significant, but hey, it’s cool seeing that I can keep up with LendingClub’s best and brightest. What I’d really like to know now is what quantitative range of estimated risk each LendingClub grade and sub-grade corresponds to, but it looks like . Does anyone know if loans grades generally correspond to certain percentage ranges like letter grades in academic classes? If not, have any ideas for better benchmarks I could use to evaluate my model’s performance? Go ahead and chime in in the comments below. that’s proprietary Previously published at https://tymick.me/blog/loan-grading-showdown

Can I Grade Loans Better Than LendingClub?

shouldn’t need any presentation, but just in case, let me give a quick intro. Y Combinator Y Combinator is an American seed accelerator launched in March 2005. It has been used to launch over 2,000 companies including Stripe, Airbnb, Twitch, DoorDash, Coinbase, Instacart, and Dropbox, to name a few. The combined valuation of the top was over $155 billion as of October, 2019. YC companies While nowadays Y Combinator includes mentors and coaches coming from the gotha of the startup elite, the mind behind the company is , a very well known tech-guru that I would credit for the contribution he made to language and for the launch of , among all his other achievements. Paul Graham Lisp Hacker News I was tracking down Y Combinator Startup School (YCSS) for a couple of years, and well, this time was the right one. The 8-week online course for startups is totally free, and from 2019 it’s held twice a year, so it’s not so hard to get a spot. All you need is to send some information about yourself and your project and grant some commitment. There aren’t specific milestones to pass, like exams or things like that but you need to attend a number of peer reviews to graduate. In the end, the value of this course it’s mostly up to yourself: if you’re willing to study the lectures, take parte in the forums and AMA sesh, interact with other students and attend the video lessons, it’s definitely worth your time. Otherwise, it’s no more useful than scrolling a  Reddit thread ( ). /r/startups The YCSS is structured in an eight-week time frame. Each week you have to submit your goals for the week and the main challenges you are facing. This is the base for the bi-weekly peer review. Then there are the lectures, videos and live sessions with the coaches. Y Combinator even created a dashboard where you can handle the totality of the course, with links to forums, special deals, Telegram group chat, video lessons, Q&A, and more. A video conference room is integrated as well, so you don't need any other tool than a laptop with an internet connection. I liked the YCSS pretty much, considering that it’s a free program. It also provides a good insight over Y Combinator method and the approach of their mentors. I was positively impressed by diversity within the classes: participants were from Indonesia, Thailand, India, North Africa, Russia, of course a lot were from the US West Coast (be ready to attend video confo at wee hours of the morning). And no matter what part of the world you come from, what background you have, what project you are working on: for better or worse we are all facing the same problems, with the same difficulties and constraints. As a side comment, I would not have minded to have a little bit of challenges here and there. A sort of final examination is something that is missing at the moment. But let’s jump back to the purpose of this post. What the hell did I learn during this process? I had many A-HA moments, but let me sum up my 10 most important takes. 1. Talk to your users: Talking to your users is essential. Perhaps the most important thing you should do. You would never ever make a product that nobody wants. Worst, to spend billions of marketing to push it down the customers’ throat. As you may know, having no market is still the first reason startups fail, accounting for more than 40% of the deaths. YCSS program is stressing a lot on that point, even asking you how many surveys you made and what feedback you got from your customers on a weekly basis. You have to become a problem seeker. You need to understand people's real problems. Don’t look for loonshot solutions addressing nobody issues. “Make something people want.” If this list would had to be a single bullet point, it would have been this one. 2. It’s not your customers’ job to give you solutions: A common mistake I see almost everyday it’s asking people what they want, instead of asking them which problems they have. “If I had asked people what they wanted, they would have said faster horses”, as Henry Ford said. It's not up to your users to tell you what they want, they just have to tell you their problems. Then it’s your job to find solutions for them. You are the product owner, not them. 3. No problem is too big to be a show-stopper: No problem is too big to have no workaround around it. During the peer reviews sessions, we discussed very different problems and issues each one of us was having at the time. And believe me, there were some hella big problems coming out. But the mood was always very good and positive. Bigger problems just need bigger workarounds. And that’s good. Let’s keep it going. 4. MVP everything: Don’t even think about spending half a year to develop a finite product. MVP, Minimum Viable Products, is your to-go word. Start small, test, develop, implement, repeat. Don’t waste away your money and your time to work on a product nobody wants or that would take so long to develop that, once it gets to the market, it would be already obsolete. 5. ABS - Always be Shipping Ship early, ship often, ship no-f*ckin-matter what. Design short product sprints, with frequent deploy. If it takes you six months to get to your first deployment in the production environment, your startup is probably going to be dead before you  get another chance to launch. 6. Forget about that “it doesn’t scale” thing: Y Combinator mentors told us this little story: it was rather common to find post-MBA folks who used to discard every single startup pitch they were evaluating by commenting “well, it doesn’t scale”. As a startup in its early stage, please, don’t deal with that. Ok your idea doesn't scale. Fine with that. It’s the starting point for everybody. First validate the concept, then worry about scaling your business. Some of the most valuable, and, well, scalable, startups on the market, started by doing things by hands or mimicking features they still had not. Who cares about that. In a world where the most ground-breaking tech is just next door, there's plenty of room to get your products to scale later on. 7. “Good product instincts” is a trap: Get rid of the  “I’m trusting my guts” BS asap. Nobody has it, and Steve Jobs is dead. Measure all your experiments and don’t rely on “instinct”, “guts”, or “good taste”. You may be surprised by the results, discovering that the ugly-looking template is converting way more than the polished, trendy one, because, well, the former has a better CTA. 8. Build a solid Company Culture: Company culture is often ignored by startups. The mood is like, “money first, culture later”. But well, there is no “later”. You have to set up culture in the very early stage of your company. It’s easy to share a culture statement among ten people, but then try to do it when the company grows to one hundred employees. Culture should work as a de-facto guideline: this is how we behave when there aren't further rules around. And culture should guide you in hiring your first employee, getting your first customer, dealing with product issues, and so on. Remember how much the simple “don’t be evil” quote meant to Google employees, driving their business and growth strategies for almost a decade. So don’t forget about culture and settle it down as soon as possible. 9. Leaders don't brag. You can’t say “I am a good leader”. That's something other people have to say. Do you want to find who’s a true leader? Ask the people around them, who they work with, but also their friends, the bartender where they have their coffee, their neighbors. Leadership is not something you turn on or off when you enter the office. And, by the same means, be careful when someone is claiming out loud their leadership skills. 10. Don’t get sad: This was from YC President Geoff Ralston’s parting advice, during the last lesson of the course. I loved that one. I never bought into the “creating startups is the coolest thing in the world” adage. Creating a biz from scratches is hard, it’s often painful. Things can go disastrously wrong in a startup and the founders feel sadder and sadder. And sadness leads to inaction. It leads to apathy. It leads to depression.  Don't do that. Don't get sad. Go, do, write code. Talk to users. Make your  product better. Always make forward progress in spite of sadness, anger, frustration. In the end it’s not the best that succeed, but the more resilient and prone to rise up failure after failure. And help others, too. If you see someone in trouble, another entrepreneur in distress, reach out to them. We live in an open ecosystem, let it thrive. End notes: "Keep it going" . Well, that’s it. Hopefully you found some inspiration and hint in my story. At least now you know a little bit better what this YCSS is about and around which pillars the course is built. If there’s something you’d like to discuss further, write me in the comments! Get back to work now. And don’t quit!

The 10 Things I’ve Learned Attending Y Combinator Startup School 2020

This post was co-written with Ben Wilcock, Product and Technical Marketing Manager for Spring at Pivotal. 🔔 A file has been uploaded! 🔔 🔔 A new user was registered! 🔔 🔔 An order was placed! 🔔 These sound like events that many parts of our application architecture might be interested in, right? For example, when an order is placed on our website, we’ll need a call to process the payment, a call to reserve inventory, and a call to begin the process of picking, packaging and shipping the product. For a single order, this isn’t too bad. Our store can make a few requests to these backend services directly and it shouldn’t introduce too much overhead.But what happens if we’re really good at selling our product? Processing 100 orders a second suddenly means our frontend is making three hundred calls per second to our backend services. If we add one more service to that—say, to report to an internal sales dashboard— now that’s four hundred calls per second. That’s a lot of overhead! What if instead, we can simply have our website alert our whole architecture at once? It can yell, “Hey! I made a sale” to our whole stack, and any component that’s interested can take the appropriate action. This means we don’t need to update our frontend as we add additional services, and our new services just need to know what to listen for. Why Spring Cloud Stream? The above is an example of an , where instead of reaching out to each service one by one, our services instead emit a change of state. If a file is uploaded, our file service can emit it out to a messaging platform, and then our Super Duper Image Resizer 3000 service can listen for that and automatically generate differently sized profile images. Pivotal’s own Richard Seroter in detail, and it’s a great read. In his blog post, Richard talks about messaging as a way of reliably delivering events to many consumers quickly and in volume. event-driven architecture wrote about this very topic He also touches on something we want to talk about today: . Spring Cloud Stream We’re big fans of both Kafka and RabbitMQ as event streaming platforms, so for this demo we’ll use Kafka. No matter which you choose to use, making it easy to produce and consume events is important for your developers. I’ve used a lot of frameworks that abstract away from the underlying message queue, but none quite as easy and flexible as Spring Cloud Stream. My teammate Ben Wilcock put together a demo that really shows just how easy it is to get up and running. Let’s take it for a spin—and to follow along, you can . download the full source code here Prepping For The Demo We only need a couple of things for our demo, which are and Docker Compose, and of course your favorite distribution of the JDK (perhaps even , which we sponsor). To keep things easy, the demo includes a Docker Compose config that will set up both Kafka and RabbitMQ, though for our purposes we’ll only be using Kafka. We can spin this up with a simple command: Docker AdoptOpenJDK docker-compose up This will read our file, download the necessary container images, run them, and configure them. After just a few moments, Kafka should be up and running and ready to go. docker-compose.yml Sending Events Our demo is made up of two Spring microservices, one to produce events and one to consume them. In our fictional scenario, the message producer will create a stream of applications for bank loans, and our processor will check if those applications should be approved or declined. Let’s start by producing some messages that will be sent to Kafka, the code for which is in the directory. loansource There are a few files of code here. The file defines a object and the file defines all the states a loan can be in. What’s interesting, though, is the file, which is what’s actually producing our messages. Loan.java loan Statuses.java LoansourceApplication.java As you can imagine, Spring and its dependencies handle a lot of the wiring up of components for us automatically. Let’s take a look at to see how this works. LoansourceApplication.java { () -> {
    String rName = names.get( Random().nextInt(names.size()));
    Long rAmount = amounts.get( Random().nextInt(amounts.size()));
    Loan loan = Loan(UUID.randomUUID().toString(), rName, rAmount);
    log.info( , loan.getStatus(), loan.getUuid(), loan.getAmount(), loan.getName()); loan;
  };
} @Bean Supplier<Loan> public supplyLoan () return new new new "{} {} for ${} for {}" return is a Java function data type. Because there is only one method that returns this type, Spring Cloud Stream knows exactly what to do next. By default, it will trigger this function once every second and send the result to the default named . What’s nice about this function method is that it only contains business logic, so you can test it using your favorite testing methods. Supplier<> @Bean MessageChannel output We could use the property in the application.properties file to explicitly declare which function bean we want to be bound to binding destinations, but for cases when you only have a single defined, this is not necessary. spring.cloud.function.definition @Bean Likewise, if we wanted to use a different poller interval, we can use the property in the file. The only question that remains is, “How does Spring know it’s Kafka we’re writing to?” For that, we take a look at our : spring.integration.poller.fixed-delay application.properties pom.xml org.springframework.cloud spring-cloud-stream-binder-kafka < > dependency < > groupId </ > groupId < > artifactId </ > artifactId </ > dependency Providing this dependency in our code tells Spring, “I’d like to send these messages to Kafka”. Since our Kafka server is listening on on the default port, we don’t need to provide any additional configuration in our file, but if that’s not the case, providing information such as hostname, port, authentication, etc. localhost application.properties we can of course do so We can run our code and activate the profile, which we’ve configured to be the profile that includes the Kafka SCS binding, and we should see it start producing messages: kafka loansource
./mvnw package spring-boot:run -DskipTests= -Pkafka cd true After a few moments, we’ll see our application start creating new loans and sending them to Kafka: ..LoansourceApplication : PENDING eff9b58-e1f1 d f1d-aa4db8dbb75a $ Donald ..LoansourceApplication : PENDING d507c06c bb a98 f85 f74af36984 $ Jacinda ..LoansourceApplication : PENDING fc86a4-d461 c ce1a258e7 $ Jacinda ..LoansourceApplication : PENDING f3756c-ea9b f-bad2 f1647188b1 $ Vladimir ..LoansourceApplication : PENDING d10f-c1c8 fe8 ce363ef56f $ Theresa 2019 -10 -15. 9 -474 -8 for 10000000 for 2019 -10 -15. -81 -4 -8 -38 for 100 for 2019 -10 -15. 19 -470 -8005 -423 for 100 for 2019 -10 -15. 33 -472 -73 for 10000 for 2019 -10 -15. 1625 -4e75 -8 -10 for 10000000 for If you prefer, you can also see the messages in your browser using . Simply point your browser to and you should see a UI that allows you to look at the messages stored in Kafka: KafDrop localhost:9000 Receiving Events We’ve got half of the equation here, but we also need something to consume and process these events. For this, we’ll look in the directory. For this half of the demo, our loan checker will observe every application and approve or decline it. If approved, an approval message will be sent to the topic otherwise, a denial message will be sent to the topic. loancheck approved declined You can extrapolate from here that other systems down the line could listen for and pick up these messages for further processing. For example, maybe a payout system listens for an approved loan to start processing it. We’ll see the code here is a little different, just pointing to different topics. We see that in , we have the annotation, meaning that all of our definitions for channel bindings are found in the class. LoanCheckApplication.java @EnableBinding(LoanProcessor.class) LoanProcessor In our file, we’ll see we define the we’re listening on is named output, matching the default topic our producer writes to. Additionally, we define two other MessageChannels that we’ll be writing to, and . For each of these, we also define which method to invoke when a message is received on those channels. LoanProcessor.java MessageChannel approved declined {
  String APPLICATIONS_IN = ;
  String APPROVED_OUT = ;
  String DECLINED_OUT = ; (APPLICATIONS_IN) ; (APPROVED_OUT) ; (DECLINED_OUT) ;
} @Component public interface LoanProcessor "output" "approved" "declined" @Input SubscribableChannel sourceOfLoanApplications () @Output MessageChannel approved () @Output MessageChannel declined () Finally, we can see how this ties into which method is invoked if we take a look at the file. We’ll see we have a method with the annotation that matches our Input we defined previously: LoanChecker.java checkAndSortLoans @StreamListener (LoanProcessor.APPLICATIONS_IN) {
  log.info( , loan.getStatus(), loan.getUuid(), loan.getAmount(), loan.getName()); (loan.getAmount() > MAX_AMOUNT) {
    loan.setStatus(Statuses.DECLINED.name());
    processor.declined().send(message(loan));
  } {
    loan.setStatus(Statuses.APPROVED.name());
    processor.approved().send(message(loan));
  }
} @StreamListener public void checkAndSortLoans (Loan loan) "{} {} for ${} for {}" if else We can start this code up much like we did our , by opening up a separate terminal and running the following: loansource loancheck
./mvnw package spring-boot:run -DskipTests= -Pkafka cd true After a few moments, we’ll start seeing our pending messages come through and then get sorted into or : approved declined ..LoanChecker : PENDING a887cf-ab5f c4-b03b cfc $ Kim ..LoanChecker : APPROVED a887cf-ab5f c4-b03b cfc $ Kim ..LoanChecker : PENDING a15f13fe-fc9a fb-b6f0 a18c0cd $ Angela ..LoanChecker : DECLINED a15f13fe-fc9a fb-b6f0 a18c0cd $ Angela 2019 -10 -15. 95 -48 -556675446 for 1000 for 2019 -10 -15. 95 -48 -556675446 for 1000 for 2019 -10 -15. -40 -24106 for 100000000 for 2019 -10 -15. -40 -24106 for 100000000 for Wrapping Up Spring Cloud Stream provides an extremely powerful abstraction for potentially complicated messaging platforms, turning the act of producing messages into just a couple lines of code. Should your infrastructure needs change and you need to migrate to a new messaging platform, not a single line of code changes other than your pom file. No matter if you’re using Kafka, RabbitMQ, or a cloud provider’s solution such as GCP Pub/Sub or Azure Event Hub, Spring Cloud Stream means it’s simple and quick to get up and running. About the Author Brian is a Principal Product Marketing Manager on the Technical Marketing team at Pivotal, with a focus on technical educational content for Pivotal customers as well as the Cloud Foundry, BOSH, and Knative communities. Prior to Pivotal, Brian worked on both the development and operations of software, with a heavy focus on Cloud Foundry and BOSH at companies in many industries including finance, entertainment and technology. He loves learning and experimenting in many fields of technology, and more importantly sharing the lessons learned along the way.

Stories, News & Mentions about LOANSHOUT

Loanshout’s Stories on HackerNoon