Mark Copy

markcopy

An Essay Concerning Humane Understanding, Volume I: Book II, Chapter XXXII. 

5 New Malware Techniques to Steal Your Crypto (2024)

আপনার ক্রিপ্টো চুরি করার জন্য 5টি নতুন ম্যালওয়্যার কৌশল (2024)

5 neue Malware-Techniken zum Stehlen Ihrer Kryptowährungen (2024)

5 nuevas técnicas de malware para robar tus criptomonedas (2024)

5 nouvelles techniques de malware pour voler vos crypto-monnaies (2024)

आपकी क्रिप्टो चोरी करने की 5 नई मैलवेयर तकनीकें (2024)

暗号資産を盗む 5 つの新しいマルウェア手法 (2024)

암호화폐를 훔치는 5가지 새로운 맬웨어 기술(2024)

5 novas técnicas de malware para roubar suas criptomoedas (2024)

5 новых вредоносных методов кражи вашей криптовалюты (2024)

Kripto Paranızı Çalmak İçin 5 Yeni Kötü Amaçlı Yazılım Tekniği (2024)

5 Kỹ thuật phần mềm độc hại mới để đánh cắp tiền điện tử của bạn (2024)

5 种窃取加密货币的新型恶意软件技术 (2024)

Welcome to other chapters of Let’s Understand Chrome V8 For performance reasons, V8 uses the compilation cache to hold the SharedFunction which is the compiler result, the cache returns the result directly once the same JavaScript code is compiled again. You may ask why the same code is compiled again. Compile first, then execute — this is the pipeline of V8. Even the exact same code, the same function, still needs to be sent to the compiler, but not always recompiled. Actually, JavaScript code may change during execution, so the first action always is that we get the JavaScript code’s hash-value from the compilation cache, return the SharedFunction if the cache is hit, and recompile the JavaScript conversely. 1. Initialization and Filling The Isolate, which is the V8 virtual machine instance, is responsible for initializing the CompilationCache, isolate::Init as shown below: bool Isolate::Init(ReadOnlyDeserializer* read_only_deserializer,
                   StartupDeserializer* startup_deserializer) {
//omit...................
#define ASSIGN_ELEMENT(CamelName, hacker_name)                  \
  isolate_addresses_[IsolateAddressId::k##CamelName##Address] = \
      reinterpret_cast<Address>(hacker_name##_address());
  FOR_EACH_ISOLATE_ADDRESS_NAME(ASSIGN_ELEMENT)
#undef ASSIGN_ELEMENT

  compilation_cache_ = new CompilationCache(this); The last line is now a CompilationCache, namely, initialization. Let’s dive into it in more depth. 1.  class V8_EXPORT_PRIVATE CompilationCache {
2.   public:
3.    MaybeHandle<SharedFunctionInfo> LookupScript(
4.        Handle<String> source, MaybeHandle<Object> name, int line_offset,
5.        int column_offset, ScriptOriginOptions resource_options,
6.        Handle<Context> native_context, LanguageMode language_mode);
7.    InfoCellPair LookupEval(Handle<String> source,
8.                            Handle<SharedFunctionInfo> outer_info,
9.                            Handle<Context> context, LanguageMode language_mode,
10.                            int position);
11.    MaybeHandle<FixedArray> LookupRegExp(Handle<String> source,
12.                                         JSRegExp::Flags flags);
13.    void PutScript(Handle<String> source, Handle<Context> native_context,
14.                   LanguageMode language_mode,
15.                   Handle<SharedFunctionInfo> function_info);
16.    void PutEval(Handle<String> source, Handle<SharedFunctionInfo> outer_info,
17.                 Handle<Context> context,
18.                 Handle<SharedFunctionInfo> function_info,
19.                 Handle<FeedbackCell> feedback_cell, int position);
20.    void PutRegExp(Handle<String> source, JSRegExp::Flags flags,
21.                   Handle<FixedArray> data);
22.    void Clear();
23.    void Remove(Handle<SharedFunctionInfo> function_info);
24.    void Iterate(RootVisitor* v);
25.    void MarkCompactPrologue();
26.    void Enable();
27.    void Disable();
28.   private:
29.    explicit CompilationCache(Isolate* isolate);
30.    ~CompilationCache() = default;
31.    base::HashMap* EagerOptimizingSet();
32.    static const int kSubCacheCount = 4;
33.    bool IsEnabled() const { return FLAG_compilation_cache && enabled_; }
34.    Isolate* isolate() const { return isolate_; }
35.    Isolate* isolate_;
36.    CompilationCacheScript script_;
37.    CompilationCacheEval eval_global_;
38.    CompilationCacheEval eval_contextual_;
39.    CompilationCacheRegExp reg_exp_;
40.    CompilationSubCache* subcaches_[kSubCacheCount];
41.    bool enabled_;
42.    friend class Isolate;
43.    DISALLOW_COPY_AND_ASSIGN(CompilationCache);
44.  }; There’re three fundamentals of the above function that are important: 1. LookupScript() at line 3. It is used to look up the cache to take out the SharedFunction that holds the bytecodes corresponding to your JavaScript code. If hit, skip compilation into the execution unit directly. 2. PutScript() at line 13. If LookupScript misses, PutScript() puts the hash corresponding to your JavaScript code and the SharedFunciton that the compiler generates into the cache before execution. 3. LookupEval and PutEval at lines 7 and 16. They are the same as LookupScript and PutScript but specialize in the JavaScript eval method because the eval needs not only SharedFunction but also context. Let’s examine LookupScript in a little more depth: 1.  MaybeHandle<SharedFunctionInfo> CompilationCache::LookupScript(
2.      Handle<String> source, MaybeHandle<Object> name, int line_offset,
3.      int column_offset, ScriptOriginOptions resource_options,
4.      Handle<Context> native_context, LanguageMode language_mode) {
5.    if (!IsEnabled()) return MaybeHandle<SharedFunctionInfo>();
6.    return script_.Lookup(source, name, line_offset, column_offset,
7.                          resource_options, native_context, language_mode);
8.  }
9.  //..................separation..........................
10.  //..................separation..........................
11.  MaybeHandle<SharedFunctionInfo> CompilationCacheScript::Lookup(
12.      Handle<String> source, MaybeHandle<Object> name, int line_offset,
13.      int column_offset, ScriptOriginOptions resource_options,
14.      Handle<Context> native_context, LanguageMode language_mode) {
15.    MaybeHandle<SharedFunctionInfo> result;
16.    {
17.      HandleScope scope(isolate());
18.      const int generation = 0;
19.      DCHECK_EQ(generations(), 1);
20.      Handle<CompilationCacheTable> table = GetTable(generation);
21.      MaybeHandle<SharedFunctionInfo> probe = CompilationCacheTable::LookupScript(
22.          table, source, native_context, language_mode);
23.      Handle<SharedFunctionInfo> function_info;
24.      if (probe.ToHandle(&function_info)) {
25.        if (HasOrigin(function_info, name, line_offset, column_offset,
26.                      resource_options)) {
27.          result = scope.CloseAndEscape(function_info);
28.        }
29.      }
30.    }
31.    Handle<SharedFunctionInfo> function_info;
32.    if (result.ToHandle(&function_info)) {
33.  #ifdef DEBUG
34.      DCHECK(HasOrigin(function_info, name, line_offset, column_offset,
35.                       resource_options));
36.  #endif
37.      isolate()->counters()->compilation_cache_hits()->Increment();
38.      LOG(isolate(), CompilationCacheEvent("hit", "script", *function_info));
39.    } else {
40.      isolate()->counters()->compilation_cache_misses()->Increment();
41.    }
42.    return result;
43.  } We can find that the CompilationCache::LookupScript is only the entrance, and the key function, as well as the important code, are all in CompilationCacheScript::Lookup(). 1. In Lookup(), the argument source is the JavaScript that you as developers are actually writing, which is going to be compiled. 2. GetTable at line 20. It is the actual table that holds data into which PutScript put. 3. CompilationCacheTable::LookupScript at line 21. Look up the table for a result in which hit means a SharedFunction or miss is null. 4. In line 25: Check if the SharedFunction is the correct one depending on Origin. Let’s take a glance at HasOrigin. 1.  // We only re-use a cached function for some script source code if the
2.  // script originates from the same place. This is to avoid issues
3.  // when reporting errors, etc.
4.  bool CompilationCacheScript::HasOrigin(Handle<SharedFunctionInfo> function_info,
5.                                         MaybeHandle<Object> maybe_name,
6.                                         int line_offset, int column_offset,
7.                                         ScriptOriginOptions resource_options) {
8.    Handle<Script> script =
9.        Handle<Script>(Script::cast(function_info->script()), isolate());
10.    Handle<Object> name;
11.    if (!maybe_name.ToHandle(&name)) {
12.      return script->name().IsUndefined(isolate());
13.    }
14.    if (line_offset != script->line_offset()) return false;
15.    if (column_offset != script->column_offset()) return false;
16.    if (!name->IsString() || !script->name().IsString()) return false;
17.    if (resource_options.Flags() != script->origin_options().Flags())
18.      return false;
19.    return String::Equals(
20.        isolate(), Handle<String>::cast(name),
21.        Handle<String>(String::cast(script->name()), isolate()));
22.  } From line 1 to line 3, the comments provide the origin check rules. Specifically, the implementation is from line 14 to line 16. Next, let’s go into PutScript. 1.  void CompilationCache::PutScript(Handle<String> source,
2.                                   Handle<Context> native_context,
3.                                   LanguageMode language_mode,
4.                                   Handle<SharedFunctionInfo> function_info) {
5.    if (!IsEnabled()) return;
6.    LOG(isolate(), CompilationCacheEvent("put", "script", *function_info));
7.    script_.Put(source, native_context, language_mode, function_info);
8.  }
9.  //.......................separation............................
10.  void CompilationCacheScript::Put(Handle<String> source,
11.                                   Handle<Context> native_context,
12.                                   LanguageMode language_mode,
13.                                   Handle<SharedFunctionInfo> function_info) {
14.    HandleScope scope(isolate());
15.    Handle<CompilationCacheTable> table = GetFirstTable();
16.    SetFirstTable(CompilationCacheTable::PutScript(table, source, native_context,
17.                                                   language_mode, function_info));
18.  } As same as LookupScript(), the CompilationCache::PutScript is entrance, all important stuff is in CompilationCacheTable::PutScript. In PutScript, I would like to inform you that the cache table will be filling. Below is the CacheTable source code: 1.  // This cache is used in two different variants. For regexp caching, it simply
2.  // maps identifying info of the regexp to the cached regexp object. Scripts and
3.  // eval code only gets cached after a second probe for the code object. To do
4.  // so, on first "put" only a hash identifying the source is entered into the
5.  // cache, mapping it to a lifetime count of the hash. On each call to Age all
6.  // such lifetimes get reduced, and removed once they reach zero. If a second put
7.  // is called while such a hash is live in the cache, the hash gets replaced by
8.  // an actual cache entry. Age also removes stale live entries from the cache.
9.  // Such entries are identified by SharedFunctionInfos pointing to either the
10.  // recompilation stub, or to "old" code. This avoids memory leaks due to
11.  // premature caching of scripts and eval strings that are never needed later.
12.  class CompilationCacheTable
13.      : public HashTable<CompilationCacheTable, CompilationCacheShape> {
14.   public:
15.    NEVER_READ_ONLY_SPACE
16.    static MaybeHandle<SharedFunctionInfo> LookupScript(
17.        Handle<CompilationCacheTable> table, Handle<String> src,
18.        Handle<Context> native_context, LanguageMode language_mode);
19.    static InfoCellPair LookupEval(Handle<CompilationCacheTable> table,
20.                                   Handle<String> src,
21.                                   Handle<SharedFunctionInfo> shared,
22.                                   Handle<Context> native_context,
23.                                   LanguageMode language_mode, int position);
24.    Handle<Object> LookupRegExp(Handle<String> source, JSRegExp::Flags flags);
25.    static Handle<CompilationCacheTable> PutScript(
26.        Handle<CompilationCacheTable> cache, Handle<String> src,
27.        Handle<Context> native_context, LanguageMode language_mode,
28.        Handle<SharedFunctionInfo> value);
29.//omit......................
30.    }; From line 1 to line 11, the comments provide more corresponding explanations enough. Let’s go back to CompilationCacheScript::Put() and ahead to line 16 which is CompilationCacheTable::PutScript(). 1.  Handle<CompilationCacheTable> CompilationCacheTable::PutScript(
2.      Handle<CompilationCacheTable> cache, Handle<String> src,
3.      Handle<Context> native_context, LanguageMode language_mode,
4.      Handle<SharedFunctionInfo> value) {
5.    Isolate* isolate = native_context->GetIsolate();
6.    Handle<SharedFunctionInfo> shared(native_context->empty_function().shared(),
7.                                      isolate);
8.    src = String::Flatten(isolate, src);
9.    StringSharedKey key(src, shared, language_mode, kNoSourcePosition);
10.    Handle<Object> k = key.AsHandle(isolate);
11.    cache = EnsureCapacity(isolate, cache, 1);
12.    int entry = cache->FindInsertionEntry(key.Hash());
13.    cache->set(EntryToIndex(entry), *k);
14.    cache->set(EntryToIndex(entry) + 1, *value);
15.    cache->ElementAdded();
16.    return cache;
17.  } I think the important thing is that PutScript binds the JavaScript source and SharedFuncation as a hash pair as well as puts it into the cache, which the implementations are from line 9 to line 15. 2. Lookup and update Let’s go into Compiler::GetSharedFunctionInfoForScript. 1.   MaybeHandle<SharedFunctionInfo> Compiler::GetSharedFunctionInfoForScript(
2.       Isolate* isolate, Handle<String> source,
3.       const Compiler::ScriptDetails& script_details,
4.       ScriptOriginOptions origin_options, v8::Extension* extension,
5.       ScriptData* cached_data, ScriptCompiler::CompileOptions compile_options,
6.       ScriptCompiler::NoCacheReason no_cache_reason, NativesFlag natives) {
7.  //omit.....................
8.       if (extension == nullptr) {
9.         bool can_consume_code_cache =
10.             compile_options == ScriptCompiler::kConsumeCodeCache;
11.         if (can_consume_code_cache) {
12.           compile_timer.set_consuming_code_cache();
13.         }
14.         maybe_result = compilation_cache->LookupScript(//lookup is here.
15.             source, script_details.name_obj, script_details.line_offset,
16.             script_details.column_offset, origin_options, isolate->native_context(),
17.             language_mode);
18.         if (!maybe_result.is_null()) {
19.           compile_timer.set_hit_isolate_cache();
20.         } else if (can_consume_code_cache) {
21.  //omit
22.           if (CodeSerializer::Deserialize(isolate, cached_data, source,
23.                                           origin_options)
24.                   .ToHandle(&inner_result) &&
25.               inner_result->is_compiled()) {
26.             is_compiled_scope = inner_result->is_compiled_scope();
27.             DCHECK(is_compiled_scope.is_compiled());
28.     		//filling is here
29.             compilation_cache->PutScript(source, isolate->native_context(),
30.                                          language_mode, inner_result);
31.             Handle<Script> script(Script::cast(inner_result->script()), isolate);
32.             maybe_result = inner_result;
33.           } else {
34.             compile_timer.set_consuming_code_cache_failed();
35.           }
36.         }
37.       }
38.       return maybe_result;
39.     } In line 14, it means that we need to compile the JavaScript code if LookupScript is missing, so we skip to line 21 to perform compilation and generate inner_result. Then at line 29, put the inner_result into the cache. We can find that the innuer_result is just the maybe_result at line 32. Figure 1 shows the call stack. Besides LookupScript and PutScript, there are also LookupEval, PutEval as well as LookupRegExp, PutRegExp in V8. The workflow of them is the same as LookupScript and PutScript, please learn by yourself. Okay, that wraps it up for this share. I’ll see you guys next time, take care! Please reach out to me if you have any issues. qq9123013 v8blink@outlook.com WeChat: Email: Also published . here

Understanding Chrome V8: Chapter 19: Compilation Cache — Make the Compiler Faster

Cloud computing has been very boring for a long time. Everything is inside the data center and you don’t even see the systems that you are working on. But not anymore. With Edge Computing, the server is no longer confined to inside the data center. It is out there touching our daily life and computing is happening in real-world and real-time. ( : IEEE Spectrum) Featured Image Source Unlike the servers inside the data center which never see the sunlight, the edge computer can be covered in snow or exposed to scorching heat in the middle of a desert. Cell station out in the desert. Source: MarcoPolisIndoor versus OutdoorCell phone tower damaged by Hurricane in Florida (Verizon) Compare with an outdoor edge server, a server hosted in the data center is like living in a sheltered environment. With outdoor edge server, it is subjected to random events which cannot be predicted e.g. natural disaster and vandalism. Hence, the design of the outdoor edge server has to be weather proof, water proof and temper proof etc. Indoor versus Outdoor Cell phone tower damaged by Hurricane in Florida (Verizon) To provide guidance to designers, a two-digit number system was created by the International Electro Technical Commission. Please see the IP Ratings guide below from NEMA Enclosures on how to decode the two-digit number. Credit: NEMA Enclosures The , or , standard 60529, sometimes interpreted as , classifies and rates the degree of protection provided by mechanical casings and against intrusion, , accidental contact, and . It is published by the (IEC). The equivalent European standard is EN 60529. IP Code Ingress Protection Code IEC International Protection Code electrical enclosures dust water International Electrotechnical Commission The standard aims to provide users more detailed information than vague marketing terms such as . For example, a cellular phone rated at IP68 is “dust resistant” and can be “immersed in 1.5 meters of freshwater for up to 30 minutes”. Similarly, an electrical socket rated IP22 is protected against insertion of fingers and will not be damaged or become unsafe during a specified test in which it is exposed to vertically or nearly vertically dripping water. IP22 or IP2X are typical minimum requirements for the design of electrical accessories for indoor use. For more details, please visit the web page below. waterproof As an example, the picture below is an IP65 design for 5G. That means the design is protected from total dust ingress and from low pressure water jets from any direction. https://www.nemaenclosures.com/blog/ingress-protection-ratings/ Credit: Supermicro What Is Inside the Edge Server An edge server can be a standalone system that handles all the processing locally without being connected to the cloud or data centers. With most of the outdoor edge servers, it will interact with the outside world. What is inside the edge server will depend on the applications. Using different add on cards and software, the edge server can support different applications. Here is an example of a 5G O-RAN Distribution Unit (DU). In a distributed 5G cloud RAN architecture the DU is collocated with the Remote Radio Unit (RRU). Inside the server, there are two add on cards. 5G O-RAN Distribution Unit. Credit Supermicro. The first card is the Intel® Ethernet 700 Series Network Adapter with Hardware-Enhanced Precision Time Protocol (PTP) which brings higher accuracy IEEE 1588 PTP synchronization signaling to the Ethernet and onto the Edge platform to provide more precise clock synchronization for 5G use cases. The second card is the Intel FPGA Programmable Acceleration Card N3000. The DU in this example will support 12 sectors of RRUs requiring a large amount of digital signal processing in real time to perform the CPRI-to-eCPRI conversion. This task can be offloaded to the FPGA card. This card accelerates network traffic up to 100Gbps to support low latency and high bandwidth tasks such as FEC acceleration at the PHY layer. (Note. CPRI stands for Common Public Radio Interface and eCPRI stands for Enhanced Common Public Radio Interface.) Now that we have figured out the hardware inside the edge server, the next step is making it work in an outdoor environment. And this is where the fun begins! Outdoor Edge Server To make the edge server ready for outdoor, the first thing we have to do is to match it with an IP65 enclosure so that it can handle the harsh outdoor environments. Since this is for telecom usage, it has to be compliant with the GR-3108-CORE and GR-487-CORE requirements. Credit: Supermicro After the enclosure is chosen, we need to figure out how to make the edge server work when it is freezing cold or under extreme heat. In this example, the operating temperature range is -40°C ~ 50°C (-40°F ~ 122°F.) To make it work, the design has a high-efficiency heat exchanger and a 300W heater. To meet the IP65 requirements, the design also has to be temper proof. The enclosure is designed to have lockable buckles, cabinet intrusion detection and self-diagnosis. Hardware cannot work without software. While it is not the scope of this article to go into details on the application software that goes inside the edge server, it is important to have robust remote management software as the edge server could be located somewhere that is hard to reach. Bringing It All Together Designing an outdoor edge server can be very challenging. However, making it all work together is the fun part. On top of the hardware and software integration, depending on the type of applications, it’ll have to pass a number of certifications before it can be deployed. And there are a lot of different variables such as how it is being connected to the network, how it is being powered and how it is being secured etc? Compare with doing a cookie-cutter server designs for the data center, the increased complexity in designing an outdoor edge server can be very daunting. But it can also be so much more fun once you get everything working. References https://www.cisco.com/c/en/us/solutions/enterprise-networks/edge-computing-architecture-5g.html https://www.intel.com/content/www/us/en/wireless-network/5g-edge-computing.html

Edge Computing Is So Fun - Part 1: Outdoor Edge Computing

Critics say it merely techwashes injustice When George Floyd was killed in May after a police officer pressed a knee into the man’s neck, protested, demanding civic leaders dismantle long-standing police practices and fundamentally reimagine public safety. millions of Americans A chorus of CEOs and companies soon joined in. Police technology providers big and small, from predictive policing company and gunshot alert firm to ’s , voiced support for racial justice. PredPol Shotspotter Amazon Jeff Bezos PredPol’s marketing manager Emmy Rey called the tragedy in Minneapolis a “breakdown of the social contract between the protector and the protected.” And she offered PredPol’s crime-predicting technology as a solution. "PredPol was founded on the audacious premise that we could help make the practice of policing better in America." Emmy Rey, PredPol marketing manager “PredPol was founded on the audacious premise that we could help make the practice of policing better in America. By ‘better’ we mean providing less bias, more transparency, and more accountability,” Rey wrote in a in June. company blog post PredpPol is one of a slate of predictive policing technologies—another, called Hunchlab, was acquired by Shotspotter, and , , and have developed their own tools, as have some police departments. They use a variety of techniques to try to prevent crime. IBM Microsoft Palantir Some enable officers to create a list of people they determine are at high risk for committing or being victims of crime. Others, like PredPol, take local crime report data and feed it into to generate predictions of future crime hotspots so that agencies patrol those areas and, ideally, prevent crime before it starts. The company has been tapped by police departments across the country, including in Los Angeles, Birmingham, Ala., and Hagerstown, Md. an algorithm “No crime means no victim, no investigation and arrest, and nobody to put through the criminal justice system,” PredPol’s Rey wrote. “Everyone in society benefits.” A map of Atlanta generated through PredPol that uses a predictive algorithm to map hotspots for potential crime. Christian Science Monitor/Getty Images Early versions of data-driven policing were used in the 1990s, but it has grown more popular and the technology more sophisticated over the last decade. By 2017, about a third of large agencies were using predictive analytics software, according to a funded by the U.S. Department of Justice. Even more planned to adopt it. report The technology was sold to law enforcement agencies as objective, but a growing body of research suggests that it may only further entrench biased policing. This summer, more than 1,400 boycotting predictive policing efforts and suggesting others do the same: “Given the structural racism and brutality in US policing, we do not believe that mathematicians should be collaborating with police departments in this manner. It is simply too easy to create a ‘scientific’ veneer for racism.” mathematicians signed a letter Some police departments have stopped using these methods after efficacy concerns were raised. “Dirty Data” Makes for Biased Predictions The technology is “perpetuating and, in some cases, concealing some of the biased practices in the police department,” said Rashida Richardson, director of policy research at , a New York University institute exploring the effect of artificial intelligence on society, which has looked at predictive policing programs around the country. AI Now “Dirty data,” she said, makes for bad predictions. 1,400+ Number of mathematicians who signed a letter urging their colleagues not to work with police on predictive policing algorithms. Police data is open to error by omission, she said. Witnesses who distrust the police may be reluctant to report shots fired, and rape or domestic violence victims may never report their abusers. Because it is based on crime reports, the data fed into the software may be less an objective picture of crime than it is a mirror reflecting a given police department’s priorities. Law enforcement may crack down on minor property crime while hardly scratching the surface of white-collar criminal enterprises, for instance. Officers may intensify drug arrests around public housing while ignoring drug use on college campuses. They looked at police departments, including in Chicago, New Orleans, and Maricopa County, Ariz., that have had problems with controversial policing practices, such as , or evidence of civil , including allegations of stop and frisk rights violations racial profiling. since “these systems are built on data produced during documented periods of flawed, racially biased, and sometimes unlawful practices and policies,” it raised “the risk of creating inaccurate, skewed, or systemically biased data.” They found that A seminal study from a few years earlier shows how this could play out with a specific type of crime—in this case, drug crime. Using a simulation, researchers Kristian Lum, now at the University of Pennsylvania, and William Isaac, now at DeepMind, an artificial intelligence company, looked at steer officers into minority neighborhoods in Oakland, Calif. whether PredPol’s predictions would isproportionately Lum and Issac had estimated drug use in the city based on the results of a national survey and found it would likely be fairly evenly distributed across the diverse city. "Over-policing imposes real costs on these communities." Researchers Kristian Lum and William Isaac in a 2016 report Yet when they fed arrest data into PredPol’s publicly available algorithm, they found the hotspots it would have predicted were not at all evenly distributed. The researchers found the algorithm would have sent police to Black neighborhoods at roughly twice the rate of White neighborhoods. The tool spat out risk areas where minorities lived—based not on drug use, but rather on places with prior police-reported drug crimes. “Over-policing imposes real costs on these communities,” Lum and Issac concluded in their 2016 report. “Increased police scrutiny and surveillance have been linked to worsening mental and physical health; and, in the extreme, additional police contact will create additional opportunities for police violence in over-policed areas.” Companies Say Claims of Bias Need Careful Attention PredPol did not respond to requests for comment. But the company’s founders, UCLA anthropology professor Jeffrey Brantingham and George Mohler, a computer science professor now at Indiana University–Purdue University Indianapolis, have done their own study, which they say indicates their methods do not result in higher minority arrest rates. In that began with acknowledging their critics, including Lum and Issac: “Though all of these studies deal with hypothetical scenarios or thought experiments, they succeed in demonstrating that careful attention needs to be paid to whether predictive policing produces biased arrests.” 2018, they published a paper Brantingham, Mohler, and their co-author, Matthew Valasik, conducted a real-life experiment with the Los Angeles Police Department. They compared the results between two groups: officers sent to PredPol-predicted crime hotspots and officers sent to hotspots picked by crime analysts, which served as a control group. They varied the groups each officer was assigned to day to day. An LAPD officer on patrol with PredPol predictive maps. The Washington Post/Getty Images Brantingham, Mohler, and Valasik found the of minority arrests was about the same whether the software or analysts made the predictions. What changed was the of arrests in the PredPol hotspots. That number was higher. rate number The increase in arrests, they wrote, is “perhaps understandable given that algorithmic crime predictions are more accurate.” Recently, Richardson and her colleagues Jason Schultz and Kate Crawford examined law enforcement agencies that use a variety of predictive programs. They acknowledged, however, that their findings “do not provide any guidance on whether arrests are themselves systemically biased” and called for “careful policy development” to prevent bias in predictive policing. Neither Brantingham nor Mohler responded to requests for comment. Robert Cheetham, the CEO of Azavea, which developed Hunchlab, said it’s important to recognize the bias in the data, and that their software sought to geographically space out predictions so police patrols wouldn’t be so concentrated on one particular neighborhood. He said Hunchlab also included information beyond crime reports, such as lighting, major events, school schedules, and locations of bars—which all can relate to crime but are not based on a single source of potentially police-biased data. The product also avoided forecasting certain crimes. Drug crimes and prostitution were out, as they are more heavily influenced by whom police choose to target. And they didn’t think the tools could be used for rape, which is often not reported and which is . unlikely to be deterred by police patrol It’s unclear if this approach has led to more equitable patrol. Some Cities Have Scrapped Programs Beyond complaints about racism, some police watchdogs have another critique: The software may not be helpful. Earlier this year, Chicago’s inspector general into the in-house predictive policing models, the Strategic Subject List and the Crime and Victimization Risk Model, which are both designed to predict an individual’s risk of being a shooting victim or offender. published an investigation An earlier investigation of the Strategic Subject List by the had found that 85 percent of people with the highest score were African American men and some had no violent criminal record whatsoever. Chicago Sun-Times The Chicago Police Department told the paper that the scores were based on a variety of factors, including the number of times the person had been shot or assaulted and arrests for violence or drugs and gang membership, and that overall the risk scores helped police prioritize whom to issue warnings or offer aid to. But the inspector general found the risk scores were unreliable and could potentially lead to harsher charging decisions by prosecutors. Chicago ended up . scrapping the program "Unfortunately, the data was kind of messed up." Mark Smith, LAPD Police Commission’s inspector general The LAPD also efforts this year after its own internal review cast doubt on its efficacy. ended one of its predictive policing The LAPD was the earliest adopter of PredPol, partnering with researchers to help develop the technology. But the police commission’s inspector general, Mark Smith, the data related to the program so unreliable it was difficult to draw conclusions. a decade ago found “Unfortunately, the data was kind of messed up,” Smith told The Markup. When they went to measure if PredPol worked, he said, “we really couldn’t tell.” A year after the report was released, the department announced it would be The announcement came mere weeks before the nation would erupt in protest after the killing of Floyd. dropping PredPol. “The cost projections of hundreds of thousands of dollars to spend on that right now versus finding that money and directing that money to other more central activities is what I have to do,” police chief Michael Moore said. Credits Originally published as " Data-Informed Predictive Policing Was Heralded As Less Biased. Is It? " with the Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.

The Truth About Less Biased Data-Informed Predictive Policing 

I Support A Free And Open Internet And We’re Making SESTA Harder Than It Needs To Be

Post-mortem diagnostics & debugging comes into the picture when you want to figure out what went wrong with your Node.js application in production. In this chapter of Node.js at Scale we will take a look at , a core project which aims to help you to do post-mortem diagnostics & debugging. node-report The diagnostics module node-report The purpose of the module is to produce a human-readable diagnostics summary file. It is meant to be used in both development and production environments. The generated report includes: JavaScript and native stack traces, heap statistics, system information, resource usage, loaded libraries. Currently node-report supports Node.js v4, v6, and v7 on AIX, Linux, MacOS, SmartOS, and Windows. Adding it to your project just takes an npm install and require: npm install node-report --save & //index.js require('node-report') Once you add node-report to your application, it will automatically Report generation can also be triggered by sending a USR2 signal to the Node.js process. listen on unhandled exceptions and fatal error events, and will trigger a report generation. Use cases of node-report Diagnostics of exceptions For the sake of simplicity, imagine you have the following endpoint in one of your applications: function myListener(request, response) { switch (request.url) { case '/exception':  throw new Error('*** exception.js: uncaught exception thrown from function myListener()'); }} This code simply throws an exception once the route handler is called. To make sure we get the diagnostics information, we have to add the module to our application, as shown previously. /exception node-report require('node-report') function my_listener(request, response) { switch (request.url) { case '/exception':  throw new Error('*** exception.js: uncaught exception thrown from function my_listener()'); } } Let’s see what happens once the endpoint gets called! Our report just got written into a file: Writing Node.js report to file: node-report.20170506.100759.20988.001.txtNode.js report completed Once you open the file, you’ll get something like this: =================== Node Report =================== 

Event: exception, location: "OnUncaughtException" Filename: node-report.20170506.100759.20988.001.txt Dump event time: 2017/05/06 10:07:59 Module load time: 2017/05/06 10:07:53 Process ID: 20988 Command line: node demo/exception.js 

Node.js version: v6.10.0 (ares: 1.10.1-DEV, http_parser: 2.7.0, icu: 58.2, modules: 48, openssl: 1.0.2k,  uv: 1.9.1, v8: 5.1.281.93, zlib: 1.2.8) 

node-report version: 2.1.2 (built against Node.js v6.10.0, 64 bit)

OS version: Darwin 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 

Machine: Gergelys-MacBook-Pro.local x86_64 — it includes.. You can think of this part as a header for your diagnostics summary the main event why the report was created, how the Node.js application was started , ( _node demo/exception.js_ ) what Node.js version was used, the host operating system, and the version of itself. node-report The stack traces The next part of the report includes the captured stack traces, both for JavaScript and the native part: =================== JavaScript Stack Trace ===================

Server.myListener (/Users/gergelyke/Development/risingstack/node-report/demo/exception.js:19:5) emitTwo (events.js:106:13) Server.emit (events.js:191:7) HTTPParser.parserOnIncoming [as onIncoming] (_http_server.js:546:12) HTTPParser.parserOnHeadersComplete (_http_common.js:99:23) In the JavaScript part, you can see.. the stack trace (which function called which one with line numbers), and where the exception occurred. In the native part, you can see the same thing — just on a lower level, in the native code of Node.js =================== Native Stack Trace =================== 

 0: [pc=0x103c0bd50] nodereport::OnUncaughtException(v8::Isolate*) [/Users/gergelyke/Development/risingstack/node-report/api.node] 

 1: [pc=0x10057d1c2] v8::internal::Isolate::Throw(v8::internal::Object*, v8::internal::MessageLocation*) [/Users/gergelyke/.nvm/versions/node/v6.10.0/bin/node]

 2: [pc=0x100708691] v8::internal::Runtime_Throw(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/gergelyke/.nvm/versions/node/v6.10.0/bin/node]

 3: [pc=0x3b67f8092a7]

 4: [pc=0x3b67f99ab41]

 5: [pc=0x3b67f921533] Heap and garbage collector metrics You can see in the heap metrics how each heap space performed during the creation of the report: new space, old space, code space, map space, large object space. These metrics include: memory size, committed memory size, capacity, used size, available size. To better understand how memory handling in Node.js works, check out the following articles: Hunting a Ghost — Finding a Memory Leak in Node.js Node.js Garbage Collection Explained =================== JavaScript Heap and GC =================== Heap space name: new_space Memory size: 2,097,152 bytes, committed memory: 2,097,152 bytes         

Capacity: 1,031,680 bytes, used: 530,736 bytes, available: 500,944 bytes Heap space name: old_space Memory size: 3,100,672 bytes, committed memory: 3,100,672 bytes 

Capacity: 2,494,136 bytes, used: 2,492,728 bytes, available: 1,408 bytes Total heap memory size: 8,425,472 bytes Total heap committed memory: 8,425,472 bytes Total used heap memory: 4,283,264 bytes Total available heap memory: 1,489,426,608 bytes Heap memory limit: 1,501,560,832 Resource usage The resource usage section includes metrics on.. CPU usage, the size of the resident set size, information on page faults, and the file system activity. =================== Resource usage =================== Process total resource usage: User mode CPU: 0.119704 secs Kernel mode CPU: 0.020466 secs Average CPU Consumption : 2.33617% Maximum resident set size: 21,965,570,048 bytes Page faults: 13 (I/O required) 5461 (no I/O required) Filesystem activity: 0 reads 3 writes The system information section includes.. environment variables, resource limits (like open files, CPU time or max memory size) and loaded libraries. Diagnostics of fatal errors The module can also help once you have a fatal error, like your application runs out of memory. node-report By default, you will get an error message something like this: <--- Last few GCs ---> 

  23249 ms: Mark-sweep 1380.3 (1420.7) -> 1380.3 (1435.7) MB, 695.6 / 0.0 ms [allocation failure] [scavenge might not succeed]. 

  24227 ms: Mark-sweep 1394.8 (1435.7) -> 1394.8 (1435.7) MB, 953.4 / 0.0 ms (+ 8.3 ms in 231 steps since start of marking, biggest step 1.2 ms) [allocation failure] [scavenge might not succeed]. On its own, this information is not that helpful. You don’t know the context, or what was the state of the application. With , it gets better. node-report First of all, in the generated post-mortem diagnostics summary you will have a more descriptive event: Event: Allocation failed - JavaScript heap out of memory, location: "MarkCompactCollector: semi-space copy, fallback in old gen" Secondly, you will get the native stack trace — that can help you to understand better why the allocation failed. Diagnostics of blocking operations Imagine you have the following loops which block your event loop. This is a performance nightmare. var list = [] for (let i = 0; i < 10000000000; i++) {  for (let j = 0; i < 1000; i++) {   list.push(new MyRecord())  }   for (let j=0; i < 1000; i++) {   list[j].id += 1   list[j].account += 2  }   for (let j = 0; i < 1000; i++) {   list.pop()  }} With you can request reports even when your process is busy, by sending the USR2 signal. Once you do that you will receive the stack trace, and you will see in a minute where your application spends time. node-report (Examples are taken for the node-report repository) The API of node-report Triggering report generation programmatically The creation of the report can also be triggered using the JavaScript API. This way your report will be saved in a file, just like when it was triggered automatically. const nodeReport = require('node-report') nodeReport.triggerReport() Getting the report as a string Using the JavaScript API, the report can also be retrieved as a string. const nodeReport = require('nodereport') const report = nodeReport.getReport() Using without automatic triggering If you don’t want to use automatic triggers you can opt-out of them by requiring the API itself — also, the file name can be specified as well: (like the fatal error or the uncaught exception) const nodeReport = require('node-report/api') nodeReport.triggerReport('name-of-the-report') Contribute If you feel like making Node.js even better, please consider joining the , where you can contribute to the module. Postmortem Diagnostics working group The Postmortem Diagnostics working group is dedicated to the support and improvement of postmortem debugging for Node.js. It seeks to elevate the role of postmortem debugging for Node, to assist in the development of techniques and tools, and to make techniques and tools known and available to Node.js users. In the next chapter of the Node.js at Scale series, we will discuss profiling Node.js Applications. If you have any questions, please let me know in the comments section below. Originally published at blog.risingstack.com on May 9, 2017. is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising & sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

Node.js Post-Mortem Diagnostics & Debugging

Reflecting on my company’s journey to Microservices, DevOps, and the Perfect Value Stream In the Manufacturing industry, the Value Stream Map (VSM) is a popular tool used to measure performance. The VSM was made famous by , and is essentially a flow chart that describes the necessary steps to produce value. Toyota For example, consider this simple VSM for a Chair Factory. An example VSM for a Chair Factory. This example illustrates the 4 steps our factory needs to produce a chair after receiving an Order: Welding, Painting, Assembling, and Inspection. There are a couple of things worth pointing out here. The steps must be done sequentially. In this example, Painting must come after Welding for each chair. The steps can be done in parallel for different work items. In this example, that means Painting and Welding can occur simultaneously, but for 2 different chairs. The total time for the VSM is the sum of all the steps. In this example, each chair requires 7 days from start to finish. The is how fast a work item can flow through the VSM. In this example, our factory could produce 1 chair every 3 days if each step always had a chair to work on. output rate Our Chair Factory working at full capacity can produce 1 chair every 3 days. The last point is especially noteworthy because it illustrates the difference between total time and . When deciding how to improve a process, the output rate explains why we should focus on the longest step (or biggest bottleneck). In this example, our workflow is stymied only by Inspection. output rate This example has been about chairs, but Value Steam Maps can also describe software delivery. Consider another VSM below; it’s similar to making chairs. An example VSM for software delivery. In this software example, Deploy is the longest step, which means the output rate is at least 3 days per Feature Request despite how fast the other steps are. In other words, this VSM is telling us that Deploy is effectively our only constraint. Both the above examples are oversimplified Value Stream Maps, but they illustrate why only the largest bottlenecks matter. And as I reflect on how my company’s architecture has evolved over the years, I’ve realized it’s these bottlenecks that have steered some of our biggest changes. The Story Of How Our Infrastructure Was Guided By Bottlenecks In the beginning, my company’s journey began with a few developers and not much else. Coordination was easy since the team was small, and there were no legacy systems to support because none existed. This provided the agility to figure out a business and product. Our Value Stream was clear. Just a couple of developers and a new Rails project Then our company grew. Users increased and product evolved, which meant new systems were needed. But managing these systems required us to take time away from development. This became our first bottleneck. Consequently, we introduced tools like and to manage our systems, and so our first big bottleneck was fixed. Capistrano Puppet More systems required required tools like Puppet and Capistrano to manage them. But our company continued to grow. More developers came, along with more deployments, hot-fixes, and configuration. Our automation helped, but many tasks still required human-assistance from our lone Sysadmin, who soon became overwhelmed. This became our second big bottleneck, and as a result more Sysadmins were hired and an Infrastructure (or ) team was formed. SRE More people and teams required an SRE team to deal with all their requests At this point, we mostly used Ruby and MySQL, and our SRE team allowed us to scale. But soon we would need JavaScript and Scala, with Kafka clusters and Cassandra databases. This turned into our next bottleneck because each new technology we introduced spawned unique requirements that hindered our SRE team with system specific procedures. Consequently, we migrated our systems onto Docker, and a , to give our Sysadmins a common interface to support our varied systems as our technology choices expanded. container orchestration (or Microservice) platform More diverse technologies required Docker to make it easier to support diverse technologies. Containers allowed us to support a breadth of technologies, but it also encouraged developers to be more involved with the operation of their systems. This meant that an increasing number of teams were asking Sysadmins for changes to their containers. Teams were building faster, and becoming more DevOps, but this dependency on the SRE team became the new bottleneck. Recognizing this trend in our organization led to the creation of a self-serve portal for developers to apply specific changes without a Sysadmin (even in production). Initially, this portal allowed users to set CPU and memory limits for their containers, the most popular request at the time, but features were added as other common requests (or bottlenecks) emerged. An early mock of our self-serve infrastructure portal for teams. This portal helped drive DevOps across our company by enabling teams to work freely, but also left us facing new challenges around cost management and access control, and needless to say, these challenges will not be the end. It’s likely that once these challenges are solved, we’ll find more bottlenecks again. The bottlenecks will never cease, but rest assured, that’s not a bad thing. The story of our infrastructure isn’t new, but what I’ve found enlightening is recognizing the endlessness of our journey. The perfect infrastructure doesn’t exist (the same can be said of any software product), but that doesn’t mean we should stop trying to improve. Reaching the finish line isn’t as important as moving forward. Continuous Bottlenecks Lead Towards Continuous Improvement Having bottlenecks to fix is a good thing. Bottlenecks represent an improvement that can be made to how we work. They guide our efforts by forcing us to constantly ask ourselves how we create value, and what that value is. In my from , Eduardo asks Mark when he’ll be done building Facebook so he can start monetizing it. Mark aptly replies that Facebook will never be done, similar to how fashion will never be done. favourite scene The Social Network The Facebook will never be done This scene resonated with me because I think it applies not only to Facebook, but to all forms of craft, whether it be software, art, music, or something else. Lessons can always be learned, growth can always be had, and in the case of software delivery, Value Streams uncover the bottlenecks that lead to the most meaningful improvements. I was told once that building software is like building railroads; the process begins with pioneering, followed by building, then finally optimization. While this comparison is sometimes useful, the finality of “optimization” felt lacking. Instead, I prefer comparing software development to Star Trek, and its mission of continual exploration, and like exploring the universe, building (and improving) is never done. We just need to decide which direction to go. The Real Difference Between Building Chairs And Writing Software Manufacturing and software share a continuous journey of improvement, but this is where the similarities end when it comes to the Value Stream Map. Manufacturing deals with the physical world. On a factory floor, you can see the work items, and the workflow itself. It might even be obvious which steps are bottlenecks and error prone. This is not so easy with software. With software, the work is electronic, and the factory floor is a room full of computers. It’s hard to see what someone is tasked, or what the workflow is. Software is invisible, which is what really differentiates it from manufacturing, and why tools like Jira are so important for this type of work. Progress is easier to spot in an assembly line than an office. Jira is a task system for software teams to describe and track work items, their progress, ownership, time spent, and other information that’s vital to understanding inefficiencies and building a meaningful Value Stream Map. These task management system are often thought of as being meant only for planning work, but a large part of their value is derived from how they chronicle the past, so we can rationalize it to make better decisions in the future. management future Creating a ticket to track a work item in Jira A lot of work has been done to understand developer productivity and its driving factors. In the age of DevOps, this has resulted in recognizing Deployment Frequency, Change Lead Time, and . However, in my experience nothing is as universally effective as simply journaling work and time. I strongly believe the benefits individuals receive from journaling and reflection, such as the ones below, applies to organizations too. other such developer metrics Learning from direct experience can be more effective if coupled with reflection . Keeping a diary to record wins and setbacks (that otherwise might slip through your memory) can provide motivation from realizing you are making progress, and help you think about how to get rid of inhibitors (even minor ones that recur) blocking your progress. The single most important thing for any software company to focus on is understanding how it creates value. Many companies get lost choosing between technologies, or organizational structures, but none of this intrinsically matter. The only thing that should matter is how value is created, and the obstacles that get in the way. And by citing these experiences (with Jira or something else), and reflecting upon them, we’re able to not only recognize the obstacles in our way, but reveal the path forward.

How Toyota Guides The Evolution Of Our Infrastructure (And Makes Me Appreciate Jira)

Stories, News & Mentions about MARKCOPY

Markcopy’s Stories on HackerNoon