Mavenoid

mavenoid

Your Definitive Guide to Lakehouse Architecture with Iceberg and MinIO

Su guía definitiva para la arquitectura Lakehouse con Iceberg y MinIO

Votre guide définitif sur l'architecture Lakehouse avec Iceberg et MinIO

आइसबर्ग और मिनिओ के साथ लेकहाउस आर्किटेक्चर के लिए आपकी निश्चित मार्गदर्शिका

Iceberg と MinIO を使用したレイクハウス アーキテクチャの決定版ガイド

Seu guia definitivo para arquitetura Lakehouse com Iceberg e MinIO

Hướng dẫn dứt khoát của bạn về Kiến trúc Lakehouse với Iceberg và MinIO

使用 Iceberg 和 MinIO 的 Lakehouse 架构权威指南

Recently, we took over a data integration project where the upstream data was delivered to Kafka. Initially, we chose the SpringBoot+Flink approach to process the incoming data (referred to as Solution 1 below). However, after facing efficiency issues in the production environment, we turned our attention to the open-source project SeaTunnel. We discovered that SeaTunnel supports Kafka as a source and started exploring and testing it. In our development environment, we experimented with over 5 million records and achieved an efficiency of around 10,000 records per second. Consequently, we abandoned Solution 1 and adopted SeaTunnel for integrated data processing (referred to as Solution 2 below). During our exploration of SeaTunnel, we identified two methods. Method 2, compared to Method 1, offers a more comprehensive solution that handles various scenarios without concerns about data misalignment caused by unexpected characters within field values. Version Information: SeaTunnel: apache-seatunnel-2.3.2-SNAPSHOT Engines: Flink 1.16.2 Zeta: Consistent with SeaTunnel Comparison Based on Solution 2, two derived methods were developed: After extensive exploration and validation in our production environment, we recommend using Method 2 of Solution 2. Now, let's dive into the main content and explore the two methods in Solution 2, allowing you to experience the magic of SeaTunnel. Solution 1: Kafka Complex JSON Parsing with Spring Boot + Flink There are many online examples available for this approach, so we won't go into much detail here. Solution 2: Kafka Complex JSON Parsing with SeaTunnel Before we proceed, let's take a look at a sample JSON data delivered to Kafka by the upstream party (part of the data has been redacted): "type": "D***",
"headers": null,
"messageSchemaId": null,
"messageSchema": null,
"message": {
    "data": {
        "LSH": "187eb13****l0214723",
        "NSRSBH": "9134****XERG56",
        "QMYC": "01*****135468",
        "QMZ": "1Zr*****UYGy%2F5bOWtrh",
        "QM_SJ": "2023-05-05 16:42:10.000000",
        "YX_BZ": "Y",
        "ZGHQ_BZ": "Y",
        "ZGHQ_SJ": "2023-06-26 16:57:17.000000",
        "SKSSQ": 202304,
        "SWJG_DM": "",
        "SWRY_DM": "00",
        "CZSJ": "2023-05-05 16:42:10.000000",
        "YNSRSBH": "9134****XERG56",
        "SJTBSJ": "2023-06-26 19:29:59.0,00",
        "SJCZBS": "I"
    },
    "beforeData": null,
    "headers": {
        "operation": "INSERT",
        "changeSequence": "12440977",
        "timestamp": "2023-06-26T19:29:59.673000",
        "streamPosition": "00****3.16",
        "transactionId": "000***0006B0002",
        "changeMask": "0FFF***FF",
        "columnMask": "0F***FFFF",
        "transactionEventCounter": 1,
        "transactionLastEvent": false
    }
} Method 1: Parsing without UDF Functions Issue: Field values may contain delimiters, such as ','. This can lead to data misalignment during storage. This method primarily utilizes various transformation plugins available in the official transform-v2. It includes the , , and implementations. Replace Split SQL SeaTunnel Script: (ybjc_qrqm.conf) job.mode = "STREAMING"
job.name = "kafka2mysql_ybjc"
execution.checkpoint.interval = 60000

source {
  Kafka {
    result_table_name = "DZFP_***_QRQM1"
    topic = "DZFP_***_QRQM"
    bootstrap.servers = "centos1:19092,centos2:19092,centos3:19092"
    schema = {
      fields {
        message =  {
            data = {
                LSH = "string",
                NSRSBH =  "string",
                QMYC =  "string",
                QMZ =  "string",
                QM_SJ =  "string",
                YX_BZ =  "string",
                ZGHQ_BZ =  "string",
                ZGHQ_SJ =  "string",
                SKSSQ =  "string",
                SWJG_DM = "string",
                SWRY_DM = "string",
                CZSJ = "string",
                YNSRSBH = "string",
                SJTBSJ = "string",
                SJCZBS = "string"
            }
        }
      }  
    }
	  start_mode = "earliest"
    kafka.config = {
      auto.offset.reset = "earliest"
      enable.auto.commit = "true"
      max.partition.fetch.bytes = "5242880"
      session.timeout.ms = "30000"
      max.poll.records = "100000"
    }
  }
}

transform {
 Replace {
    source_table_name = "DZFP_***_QRQM1"
    result_table_name = "DZFP_***_QRQM2"
    replace_field = "message"
    pattern = "[["
    replacement = ""
  }
  Replace {
    source_table_name = "DZFP_***_QRQM2"
    result_table_name = "DZFP_***_QRQM3"
    replace_field = "message"
    pattern = "]]"
    replacement = ""
  }
  
  Split {
    source_table_name = "DZFP_***_QRQM3"
    result_table_name = "DZFP_***_QRQM4"
    separator = ","
    split_field = "message"
    output_fields = [zwf1,zwf2,zwf3,zwf4,zwf5,nsrsbh,qmyc,qmz,qm_sj,yx_bz,zghq_bz,zghq_sj,skssq,swjg_dm ,swry_dm ,czsj ,ynsrsbh ,sjtbsj ,sjczbs]
  }
  
  sql{
   source_table_name = "DZFP_***_QRQM4"
   query = "select replace(zwf5 ,'fields=[','') as lsh,nsrsbh,trim(qmyc) as qmyc,qmz,qm_sj,yx_bz, zghq_bz,zghq_sj,skssq,swjg_dm ,swry_dm ,czsj ,ynsrsbh ,sjtbsj ,replace(sjczbs,']}]}','') as sjczbs  from DZFP_DZDZ_QRPT_YWRZ_QRQM4 where skssq <> ' null'"
   result_table_name = "DZFP_***_QRQM5"
  }
  
}

sink {
	Console {
		source_table_name = "DZFP_***_QRQM5"
	}
	
    jdbc {
	    source_table_name = "DZFP_***_QRQM5"
        url = "jdbc:mysql://localhost:3306/dbname?serverTimezone=GMT%2b8"
        driver = "com.mysql.cj.jdbc.Driver"
        user = "user"
        password = "pwd"
        batch_size = 200000
        database = "dbname"
        table = "tablename"
        generate_sink_sql = true
        primary_keys = ["nsrsbh","skssq"]
    } 
} Data can be successfully written: Kafka Source Data: Target Data in TiDB: However, when we simulate sending data to Kafka, specifically setting a ',' as part of the 'SJTBSJ' field: Original value: Modified value: 2023-06-26 19:29:59.000 2023-06-26 19:29:59.0,00 Command to produce data to the topic: kafka-console-producer.sh --topic DZFP_***_QRQM --broker-list centos1:19092,centos2:19092,centos3:19092 Data sent: "type": "D***",
"headers": null,
"messageSchemaId": null,
"messageSchema": null,
"message": {
    "data": {
        "LSH": "187eb13****l0214723",
        "NSRSBH": "9134****XERG56",
        "QMYC": "01*****135468",
        "QMZ": "1Zr*****UYGy%2F5bOWtrh",
        "QM_SJ": "2023-05-05 16:42:10.000000",
        "YX_BZ": "Y",
        "ZGHQ_BZ": "Y",
        "ZGHQ_SJ": "2023-06-26 16:57:17.000000",
        "SKSSQ": 202304,
        "SWJG_DM": "",
        "SWRY_DM": "00",
        "CZSJ": "2023-05-05 16:42:10.000000",
        "YNSRSBH": "9134****XERG56",
        "SJTBSJ": "2023-06-26 19:29:59.0,00",
        "SJCZBS": "I"
    },
    "beforeData": null,
    "headers": {
        "operation": "INSERT",
        "changeSequence": "12440977",
        "timestamp": "2023-06-26T19:29:59.673000",
        "streamPosition": "00****3.16",
        "transactionId": "000***0006B0002",
        "changeMask": "0FFF***FF",
        "columnMask": "0F***FFFF",
        "transactionEventCounter": 1,
        "transactionLastEvent": false
    }
} After writing the data, we notice that the data is misaligned: Conclusion: This issue can still occur in production environments when fields contain delimiters like commas in addresses or remarks. Therefore, this approach is not recommended as it poses significant risks to the data. It can be used for handling simple data as an alternative approach. Method 2: Parsing with UDF Functions This method extends SeaTunnel's capabilities using UDF functions ( ), enabling nested JSON source data parsing from Kafka. It simplifies the ST script configuration significantly. https://seatunnel.apache.org/docs/2.3.2/transform-v2/sql-udf ST Script: (ybjc_qrqm_yh.conf) execution.parallelism = 5
job.mode = "STREAMING"
job.name = "kafka2mysql_ybjc_yh"
execution.checkpoint.interval = 60000
}

source {
    Kafka {
        result_table_name = "DZFP_***_QRQM1"
        topic = "DZFP_***_QRQM"
        bootstrap.servers = "centos1:19092,centos2:19092,centos3:19092"
        schema = {
        fields {
             message =  {
                data = "map<string,string>"
             }
        }  
        }
        start_mode = "earliest"
        kafka.config = {
        auto.offset.reset = "earliest"
        enable.auto.commit = "true"
        max.partition.fetch.bytes = "5242880"
        session.timeout.ms = "30000"
        max.poll.records = "100000"
        }
    }
}
transform {
    sql{
        source_table_name = "DZFP_***_QRQM1"
        result_table_name = "DZFP_***_QRQM2"
        query = "select qdmx(message,'lsh') as lsh,qdmx(message,'nsrsbh') as nsrsbh,qdmx(message,'qmyc') as qmyc,qdmx(message,'qmz') as qmz,qdmx(message,'qm_sj') as qm_sj,qdmx(message,'yx_bz') as yx_bz,qdmx(message,'zghq_bz') as zghq_bz,qdmx(message,'zghq_sj') as zghq_sj,qdmx(message,'skssq') as skssq,qdmx(message,'swjg_dm') as swjg_dm,qdmx(message,'swry_dm') as swry_dm,qdmx(message,'czsj') as czsj,qdmx(message,'ynsrsbh') as ynsrsbh, qdmx(message,'sjtbsj') as sjtbsj,qdmx(message,'sjczbs') as sjczbs  from  DZFP_DZDZ_QRPT_YWRZ_QRQM1"
    }
}

    
sink {
    Console {
        source_table_name = "DZFP_***_QRQM2"
    }
    jdbc {
        source_table_name = "DZFP_***_QRQM2"
        url = "jdbc:mysql://localhost:3306/dbname?serverTimezone=GMT%2b8"
        driver = "com.mysql.cj.jdbc.Driver"
        user = "user"
        password = "pwd"
        batch_size = 200000
        database = "dbname"
        table = "tablename"
        generate_sink_sql = true
        primary_keys = ["nsrsbh","skssq"]
    } 
} Executing the script and checking the results, we can see that the data is not misaligned and remains in the original field ('sjtbsj'): The specific UDF function is implemented as follows. Maven dependencies are as follows: <dependency>
    <groupId>org.apache.seatunnel</groupId>
    <artifactId>seatunnel-transforms-v2</artifactId>
    <version>2.3.2</version>
    <scope>provided</scope>
  </dependency>

  <dependency>
    <groupId>org.apache.seatunnel</groupId>
    <artifactId>seatunnel-api</artifactId>
    <version>2.3.2</version>
  </dependency>

  <dependency>
    <groupId>cn.hutool</groupId>
    <artifactId>hutool-all</artifactId>
    <version>5.8.20</version>
  </dependency>

  <dependency>
    <groupId>com.google.auto.service</groupId>
    <artifactId>auto-service-annotations</artifactId>
    <version>1.1.1</version>
    <optional>true</optional>
    <scope>compile</scope>
  </dependency>

  <dependency>
    <groupId>com.google.auto.service</groupId>
    <artifactId>auto-service</artifactId>
    <version>1.1.1</version>
    <optional>true</optional>
    <scope>compile</scope>
  </dependency>
</dependencies> UDF-specific implementation in Java: import cn.hutool.core.util.StrUtil;
import cn.hutool.json.JSONObject;
import cn.hutool.json.JSONUtil;
import com.google.auto.service.AutoService;
import org.apache.seatunnel.api.table.type.BasicType;
import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
import org.apache.seatunnel.transform.sql.zeta.ZetaUDF;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

@AutoService(ZetaUDF.class)
public class QdmxUDF implements ZetaUDF {

    @Override
    public String functionName() {
        return "QDMX";
    }

    @Override
    public SeaTunnelDataType<?> resultType(List<SeaTunnelDataType<?>> list) {
        return BasicType.STRING_TYPE;
    }

    // list Example of parameters: (which is the data parsed by Kafka)
    //SeaTunnelRow{tableId=, kind=+I, fields=[{key1=value1,key2=value2,.....}]}
    @Override
    public Object evaluate(List<Object> list) {
        String str = list.get(0).toString();
        //1 Remove the prefix
        str = StrUtil.replace(str, "SeaTunnelRow{tableId=, kind=+I, fields=[{", "");
        //2 Remove the suffix
        str = StrUtil.sub(str, -3, 0);
        // 3 build Map key value
        Map<String, String> map = parseToMap(str);
        if ("null".equals(map.get(list.get(1).toString())))
            return "";
        // 4 return the value of the key
        return map.get(list.get(1).toString());
    }

    public static Map<String, String> parseToMap(String input) {
        Map<String, String> map = new HashMap<>();
        // Remove the curly brackets and remove them during the string stage.
        // input = input.replaceAll("[{}]", "");
        // Split key-value pairs
        String[] pairs = input.split(", ");

        for (String pair : pairs) {
            String[] keyValue = pair.split("=");
            if (keyValue.length == 2) {
                String key = keyValue[0].trim().toLowerCase();
                String value = keyValue[1].trim();
                map.put(key, value);
            }
        }
        return map;
    }
} Then, for packaging, use the following command: mvn -T 8 clean install -DskipTests -Dcheckstyle.skip -Dmaven.javadoc.skip=true Check the directory to see if the @AutoService annotation generates the corresponding SPI interface. META-INF/services If it is as shown below, then the packaging is successful! If not, the packaging fails, and the UDF function cannot be used. You can refer to my packaging plugin: <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-site-plugin</artifactId>
      <version>3.7</version>
      <dependencies>
        <dependency>
          <groupId>org.apache.maven.doxia</groupId>
          <artifactId>doxia-site-renderer</artifactId>
          <version>1.8</version>
        </dependency>
      </dependencies>
    </plugin>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.8.1</version>
      <configuration>
        <annotationProcessorPaths>
          <path>
            <groupId>com.google.auto.service</groupId>
            <artifactId>auto-service</artifactId>
            <version>1.1.1</version>
          </path>
        </annotationProcessorPaths>
      </configuration>
    </plugin>
  </plugins>
</build> Finally, place the packaged JAR file in the $ directory. Since my UDF function depends on a third-party JAR, it should be uploaded together. If using the Zeta cluster, you need to restart the Zeta cluster for it to take effect. For other engines, it will take effect in real time. {SEATUNNEL_HOME}/lib The successful upload will look like this: Note: The "hutool-all" JAR file can be included in the project. I uploaded two for convenience. java_study In conclusion, it is recommended to to parse the nested JSON data from Kafka sources. use the extension of UDF functions Also published . here

Resolving Complex JSON in Kafka Source Using Apache SeaTunnel

Welcome to other chapters of Let’s Understand Chrome V8 Dispatch is responsible for scheduling bytecode, which is equivalent to registering EIP++, and jumps to the next bytecode. Dispatch consists of two parts, one is a dispatch table and the other is the physical register. The table is an array that contains all bytecode addresses. V8 uses the physical register to dispatch bytecode for getting greater efficiency. 1. Dispatch Table The dispatch table is an array of pointers, which is used to store the address of the bytecode processing program. The array type is Code, and the Code Class is below: class Code: public HeapObject {
    public: NEVER_READ_ONLY_SPACE
    using Flags = uint32_t;
    #define CODE_KIND_LIST(V)\
    V(OPTIMIZED_FUNCTION)\
    V(BYTECODE_HANDLER)\
    V(STUB)\
    V(BUILTIN)\
    V(REGEXP)\
    V(WASM_FUNCTION)\
    V(WASM_TO_CAPI_FUNCTION)\
    V(WASM_TO_JS_FUNCTION)\
    V(JS_TO_WASM_FUNCTION)\
    V(JS_TO_JS_FUNCTION)\
    V(WASM_INTERPRETER_ENTRY)\
    V(C_WASM_ENTRY)
    enum Kind {
      #define DEFINE_CODE_KIND_ENUM(name) name,
        CODE_KIND_LIST(DEFINE_CODE_KIND_ENUM)
      #undef DEFINE_CODE_KIND_ENUM
      NUMBER_OF_KINDS
    };
    static
    const char * Kind2String(Kind kind);
    #ifdef ENABLE_DISASSEMBLER
    const char * GetName(Isolate * isolate) const;
    V8_EXPORT_PRIVATE void Disassemble(const char * name, std::ostream & os,
      Address current_pc = kNullAddress);
    #endif
    //................omit................
  }; : In the isolate, there is another pointer array that is the same type as the dispatch table. It holds all the Builtins addresses, which can confuse beginners easily. Note The dispatch table is maintained by BuildWithMacroAssembler, code is below: Code BuildWithMacroAssembler(Isolate * isolate, int32_t builtin_index,
    MacroAssemblerGenerator generator,
    const char * s_name) {
    HandleScope scope(isolate);
    // Canonicalize handles, so that we can share constant pool entries pointing
    // to code targets without dereferencing their handles.
    CanonicalHandleScope canonical(isolate);
    constexpr int kBufferSize = 32 * KB;
    byte buffer[kBufferSize];
    MacroAssembler masm(isolate, BuiltinAssemblerOptions(isolate, builtin_index),
      CodeObjectRequired::kYes,
      ExternalAssemblerBuffer(buffer, kBufferSize));
    masm.set_builtin_index(builtin_index);
    DCHECK(!masm.has_frame());
    generator( & masm);
    int handler_table_offset = 0;
    // JSEntry builtins are a special case and need to generate a handler table.
    DCHECK_EQ(Builtins::KindOf(Builtins::kJSEntry), Builtins::ASM);
    DCHECK_EQ(Builtins::KindOf(Builtins::kJSConstructEntry), Builtins::ASM);
    DCHECK_EQ(Builtins::KindOf(Builtins::kJSRunMicrotasksEntry), Builtins::ASM);
    if (Builtins::IsJSEntryVariant(builtin_index)) {
      handler_table_offset = HandlerTable::EmitReturnTableStart( & masm);
      HandlerTable::EmitReturnEntry( &
        masm, 0, isolate -> builtins() -> js_entry_handler_offset());
    }
    //.....................................................
    //................omit.............................
  } You can see regarding how to debug the BuildWithMacroAssembler. When the value of the parameter builtin_index is 65, the function pointer generator points to which is responsible for maintaining the Dispatch table, code is below: Chapter 9 Generate_InterpreterEnterBytecodeDispatch static void Generate_InterpreterEnterBytecode(MacroAssembler * masm) {
    // Set the return address to the correct point in the interpreter entry
    // trampoline.
    Label builtin_trampoline, trampoline_loaded;
    Smi interpreter_entry_return_pc_offset(
      masm -> isolate() -> heap() -> interpreter_entry_return_pc_offset());
    DCHECK_NE(interpreter_entry_return_pc_offset, Smi::kZero);
    // If the SFI function_data is an InterpreterData, the function will have a
    // custom copy of the interpreter entry trampoline for profiling. If so,
    // get the custom trampoline, otherwise grab the entry address of the global
    // trampoline.
    __ movq(rbx, Operand(rbp, StandardFrameConstants::kFunctionOffset));
    __ LoadTaggedPointerField(
      rbx, FieldOperand(rbx, JSFunction::kSharedFunctionInfoOffset));
    __ LoadTaggedPointerField(
      rbx, FieldOperand(rbx, SharedFunctionInfo::kFunctionDataOffset));
    __ CmpObjectType(rbx, INTERPRETER_DATA_TYPE, kScratchRegister);
    __ j(not_equal, & builtin_trampoline, Label::kNear);
    __ movq(rbx,
      FieldOperand(rbx, InterpreterData::kInterpreterTrampolineOffset));
    __ addq(rbx, Immediate(Code::kHeaderSize - kHeapObjectTag));
    __ jmp( & trampoline_loaded, Label::kNear);
    __ bind( & builtin_trampoline);
    // TODO(jgruber): Replace this by a lookup in the builtin entry table.
    __ movq(rbx,
      __ ExternalReferenceAsOperand(
        ExternalReference::
        address_of_interpreter_entry_trampoline_instruction_start(
          masm -> isolate()),
        kScratchRegister));
    __ bind( & trampoline_loaded);
    __ addq(rbx, Immediate(interpreter_entry_return_pc_offset.value()));
    __ Push(rbx);
    // Initialize dispatch table register.
    __ Move(
      kInterpreterDispatchTableRegister,
      ExternalReference::interpreter_dispatch_table_address(masm -> isolate()));
    // Get the bytecode array pointer from the frame.
    __ movq(kInterpreterBytecodeArrayRegister,
      Operand(rbp, InterpreterFrameConstants::kBytecodeArrayFromFp));
    if (FLAG_debug_code) {
      // Check function data field is actually a BytecodeArray object.
      __ AssertNotSmi(kInterpreterBytecodeArrayRegister);
      __ CmpObjectType(kInterpreterBytecodeArrayRegister, BYTECODE_ARRAY_TYPE,
        rbx);
      __ Assert(
        equal,
        AbortReason::kFunctionDataShouldBeBytecodeArrayOnInterpreterEntry);
    }
    // Get the target bytecode offset from the frame.
    __ movq(kInterpreterBytecodeOffsetRegister,
      Operand(rbp, InterpreterFrameConstants::kBytecodeOffsetFromFp));
    __ SmiUntag(kInterpreterBytecodeOffsetRegister,
      kInterpreterBytecodeOffsetRegister);
    // Dispatch to the target bytecode.
    __ movzxbq(r11, Operand(kInterpreterBytecodeArrayRegister,
      kInterpreterBytecodeOffsetRegister, times_1, 0));
    __ movq(kJavaScriptCallCodeStartRegister,
      Operand(kInterpreterDispatchTableRegister, r11,
        times_system_pointer_size, 0));
    __ jmp(kJavaScriptCallCodeStartRegister);
  }
  //================================separation=========================
  void Builtins::Generate_InterpreterEnterBytecodeDispatch(MacroAssembler * masm) {
    Generate_InterpreterEnterBytecode(masm);
  } In the above code, the is the entry point of bytecode dispatch, and the is the key code that implements the bytecode dispatch. Here are two important concepts: Generate_InterpreterEnterBytecodeDispatch Generate_InterpreterEnterBytecode The is a Builtin and its index is 65. (The index may be dissimilar in different V8 versions) Generate_InterpreterEnterBytecodeDispatch V8 uses a physical register to manage the dispatch table, the name of the register is . Using the physical register can avoid the push and pop of the dispatch table frequently, to improve efficiency. kInterpreterDispatchTableRegister Here are the key lines of code： Line 35, loads the dispatch table to the A register. Line 37 of code, take out the table address from the isolate. The implementation of A is in the below three functions. (1) ExternalReference ExternalReference::interpreter_dispatch_table_address(
  Isolate * isolate) {
  return ExternalReference(isolate -> interpreter() -> dispatch_table_address());
}
interpreter::Interpreter * interpreter() const {
  return interpreter_;
}
Address dispatch_table_address() {
  return reinterpret_cast < Address > ( & dispatch_table_[0]);
} Line 39 takes out the bytecode array from the stack. (2) Line 51 takes out the offset of the target bytecode from the bytecode array. (3) Lines 56 and 58 calculate the target bytecode address and store the address in kJavaScriptCallCodeStartRegister. (4) Line 61 jumps to register kJavaScriptCallCodeStartRegister and executes bytecode. Figure 1 is the call stack. (5) 2. TailCall The Dispatch() is the last step in a bytecode, so its alias is the tail call. Let’s look at a few bytecodes： IGNITION_HANDLER(StaGlobal, InterpreterAssembler) {
    TNode < Context > context = GetContext();
    //omit..............
    Goto( & end);
    Bind( & no_feedback);
    CallRuntime(Runtime::kStoreGlobalICNoFeedback_Miss, context, value, name);
    Goto( & end);
    Bind( & end);
    Dispatch(); // !!! this is the tail call.
  }
  IGNITION_HANDLER(LdaContextSlot, InterpreterAssembler) {
    TNode < Context > context = CAST(LoadRegisterAtOperandIndex(0));
    TNode < IntPtrT > slot_index = Signed(BytecodeOperandIdx(1));
    TNode < Uint32T > depth = BytecodeOperandUImm(2);
    TNode < Context > slot_context = GetContextAtDepth(context, depth);
    TNode < Object > result = LoadContextElement(slot_context, slot_index);
    SetAccumulator(result);
    Dispatch(); //!!! here also is.
  }
  IGNITION_HANDLER(LdaImmutableContextSlot, InterpreterAssembler) {
    TNode < Context > context = CAST(LoadRegisterAtOperandIndex(0));
    TNode < IntPtrT > slot_index = Signed(BytecodeOperandIdx(1));
    TNode < Uint32T > depth = BytecodeOperandUImm(2);
    TNode < Context > slot_context = GetContextAtDepth(context, depth);
    TNode < Object > result = LoadContextElement(slot_context, slot_index);
    SetAccumulator(result);
    Dispatch(); //!!! here 
  } The above three bytecodes all end with Dispatch(). Below is the code of Dispatch: void InterpreterAssembler::Dispatch() {
    Comment("========= Dispatch");
    DCHECK_IMPLIES(Bytecodes::MakesCallAlongCriticalPath(bytecode_), made_call_);
    TNode < IntPtrT > target_offset = Advance();
    TNode < WordT > target_bytecode = LoadBytecode(target_offset);
    if (Bytecodes::IsStarLookahead(bytecode_, operand_scale_)) {
      target_bytecode = StarDispatchLookahead(target_bytecode);
    }
    DispatchToBytecode(target_bytecode, BytecodeOffset());
  } The 2nd line is the comment function. When debugging, we can use its output information to locate the execution position. The 4th line takes out the offset of the target(next) bytecode. The 5th line loads the target bytecode. The 9th line calls which calls eventually. DspatchToBytecode() DispatchToBytecodeHandlerEntry() void InterpreterAssembler::DispatchToBytecodeHandlerEntry(
  TNode < RawPtrT > handler_entry, TNode < IntPtrT > bytecode_offset) {
  // Propagate speculation poisoning.
  TNode < RawPtrT > poisoned_handler_entry =
    UncheckedCast < RawPtrT > (WordPoisonOnSpeculation(handler_entry));
  TailCallBytecodeDispatch(InterpreterDispatchDescriptor {},
    poisoned_handler_entry, GetAccumulatorUnchecked(),
    bytecode_offset, BytecodeArrayTaggedPointer(),
    DispatchTablePointer());
} In , parameter 1 is the target bytecode and 2 is the offset of the bytecode. Parameter 1 is used to load the bytecode handler and 2 is used to load operands. Then call , which is below: DispatchToBytecodeHandlerEntry TailCallBytecodeDispatch template < class...TArgs >
    void CodeAssembler::TailCallBytecodeDispatch(
      const CallInterfaceDescriptor & descriptor, TNode < RawPtrT > target,
        TArgs...args) {
      DCHECK_EQ(descriptor.GetParameterCount(), sizeof...(args));
      auto call_descriptor = Linkage::GetBytecodeDispatchCallDescriptor(
        zone(), descriptor, descriptor.GetStackParameterCount());
      Node * nodes[] = {
        target,
        args...
      };
      CHECK_EQ(descriptor.GetParameterCount() + 1, arraysize(nodes));
      raw_assembler() -> TailCallN(call_descriptor, arraysize(nodes), nodes);
    } In the above code, the 5th line creates the descriptor, as below: CallDescriptor * Linkage::GetBytecodeDispatchCallDescriptor(
    Zone * zone,
    const CallInterfaceDescriptor & descriptor,
      int stack_parameter_count) {
    const int register_parameter_count = descriptor.GetRegisterParameterCount();
    const int parameter_count = register_parameter_count + stack_parameter_count;
    DCHECK_EQ(descriptor.GetReturnCount(), 1);
    LocationSignature::Builder locations(zone, 1, parameter_count);
    locations.AddReturn(regloc(kReturnRegister0, descriptor.GetReturnType(0)));
    for (int i = 0; i < parameter_count; i++) {
      if (i < register_parameter_count) {
        // The first parameters go in registers.
        Register reg = descriptor.GetRegisterParameter(i);
        MachineType type = descriptor.GetParameterType(i);
        locations.AddParam(regloc(reg, type));
      } else {
        int stack_slot = i - register_parameter_count - stack_parameter_count;
        locations.AddParam(LinkageLocation::ForCallerFrameSlot(
          stack_slot, MachineType::AnyTagged()));
      }
    }
    MachineType target_type = MachineType::Pointer();
    LinkageLocation target_loc = LinkageLocation::ForAnyRegister(target_type);
    const CallDescriptor::Flags kFlags =
      CallDescriptor::kCanUseRoots | CallDescriptor::kFixedTargetRegister;
    return new(zone) CallDescriptor( // --
      CallDescriptor::kCallAddress, // kind
      target_type, // target MachineType
      target_loc, // target location
      locations.Build(), // location_sig
      stack_parameter_count, // stack_parameter_count
      Operator::kNoProperties, // properties
      kNoCalleeSaved, // callee-saved registers
      kNoCalleeSaved, // callee-saved fp
      kFlags, // flags
      descriptor.DebugName());
  } In , parameter 2 is the target bytecode, and parameter 3 is the stack parameter count. This function is used to apply for register resources and create a . The gives what parameters to supply when executing a bytecode. The comments on lines 24 to 25 give its layout. GetBytecodeDispatchCallDescriptor CallDescriptor CallDescriptor In , the on the 9th line is below, we see that it’s parameter 1 is . TailCallBytecodeDispatch TailCallN() CallDescriptor void RawMachineAssembler::TailCallN(CallDescriptor * call_descriptor,
    int input_count, Node *
    const * inputs) {
    // +1 is for target.
    DCHECK_EQ(input_count, call_descriptor -> ParameterCount() + 1);
    Node * tail_call =
      MakeNode(common() -> TailCall(call_descriptor), input_count, inputs);
    schedule() -> AddTailCall(CurrentBlock(), tail_call);
    current_block_ = nullptr;
  } The 5th line generates a node. In the 7th line adds the node to the end of the current basic block. So far, the dispatch is completed and the next bytecode is started to execute. Figure 2 shows the function call stack. AddTailCall() Okay, that wraps it up for this share. I’ll see you guys next time, take care! Please reach out to me if you have any issues. : qq9123013 : v8blink@outlook.com WeChat Email Also published . here

Let's Understand Chrome V8 — Chapter 11: Bytecode Dispatch 

I love 3d effects on the internet, and they're increasingly all over the place. They can add an extra dimension to a website that helps to catch a user's attention. I have previously covered 3d effects with the WebGL cards I made, but in this guide, I wanted to create a morphing sphere with a cool, wireframe background. Through this tutorial, I'll also help you understand the basics of how 3d effects work on the web. Let's get started. Here is the demo Creating a 3D Morphing Sphere The tool I use for accessing WebGL functionality in the browser is called three.js. It's a package that simplifies the process of doing 3d work in the browser - and to do this, it uses . We'll append this through code later. canvas It's important to understand that three.js simply gives us an interface to work with WebGL, which is an API for rendering 2d and 3d objects on the web. that's why we'll import three.js next. You can do this through . The two things we're going to want to install here are as follows: npm npm i three
npm i open-simplex-noise
npm install three-orbitcontrols Import these into your code once they're installed. Since I did my demo on codepen, I imported them using skypack. These three functions will let us do three (pun not intended) things: use 3d shapes on the web (three) control the camera (three-orbitcontrols) create noise and randomness (open-simplex-noise) import * as THREE from "https://cdn.skypack.dev/three@0.133.1";
import { OrbitControls } from "https://cdn.skypack.dev/three@0.133.1/examples/jsm/controls/OrbitControls.js";
import openSimplexNoise from 'https://cdn.skypack.dev/open-simplex-noise'; Setting up our scene for 3d objects in three.js After we've imported our packages, we want to do a few things: , for our 3d objects to sit on create a new scene , so we can look at our scene create a camera , and set its size so we don't get weird fuzzy shapes create a renderer , so we can click and drag our object and move it around add in our orbital controls // Scene
let scene = new THREE.Scene();
// Camera
let camera = new THREE.PerspectiveCamera( 75, innerWidth / innerHeight, 0.1, 1000 );
camera.position.set(1.5, -0.5, 6);
// Renderer
let renderer = new THREE.WebGLRenderer({antialias: true, alpha: true});
renderer.setSize( innerWidth, innerHeight );
// Append our renderer to the webpage. Basically, this appends the `canvas` to our webpage.
document.body.appendChild( renderer.domElement );

new OrbitControls(camera, renderer.domElement); Now, the fun can begin, and we can start adding in our 3d shapes. Adding 3d shapes to our scene with three.js First up, is our sphere. Each object in three.js consists of two parts - the , which is the vertices and triangles that make up the sphere, and the , which is the colors, patterns, and features of those vertices. geometry mesh Since we want to manipulate all of our vertices ultimately, I am going to store them all separately in a array too. We will use the built in function to store sets of 3d coordinates in three.js. positionData Vector3 // Create our geometry
let sphereGeometry = new THREE.SphereGeometry(1.5, 100, 100);

// This section is about accessing our geometry vertices and their locations
sphereGeometry.positionData = [];
let v3 = new THREE.Vector3();
for (let i = 0; i < sphereGeometry.attributes.position.count; i++){
    v3.fromBufferAttribute(sphereGeometry.attributes.position, i);
    sphereGeometry.positionData.push(v3.clone());
}

// A `normal` material uses the coordinates of an object to calculate its color
let sphereMesh = new THREE.MeshNormalMaterial();

// Combine both, and add it to the scene.
let sphere = new THREE.Mesh(sphereGeometry, sphereMesh);
scene.add(sphere); Using Custom Shaders Instead Now, caveat here, I decided to make my sphere a little more customizable, and to do that, I used . So, when we call , it actually does something a bit unusual for the web. It uses something called shaders to calculate the color of each vertex. one shaders MeshNormalMaterial There are two types of shaders, , which is essentially the colors of the object, and , which is the position of the vertices on that shape. These shaders are written in or - so not Javascript. I'm not going to go into detail about how this language works, but it's a little bit more like C than Javascript. fragment vertex GLSL OpenGL Shading Language The fundamentals are: Instead of using , we can use , and build our own shaders. MeshNormalMaterial ShaderMaterial We will use shaders - so the same effect will occur, but having them in our code means we can update it later - for example, change the colors. Normal Material We can pass Javascript variables into the shader in real time using , which are a special type of variable in GLSL. uniforms That means we define our GLSL in the HTML, and pull it in with a Javascript selector. : I haven't made any real changes to these shaders compared to - the only difference is I am passing in a color as a . That means we can change this value from Javascript if we want. I'll only show the fragment shader here, but both can be found in the . that I define - that's the variable we will be using from our Javascript! Note MeshNormalMaterial uniform HTML section on codepen Notice uniform vec3 colorA Ultimately, shaders make a manipulation for every pixel of our 3D object based on what we tell them to do. They are computationally expensive, but can create some pretty cool effects. <script id="fragment" type="text/glsl">
uniform vec3 colorA;
#define NORMAL
#if defined( FLAT_SHADED ) || defined( USE_BUMPMAP ) || defined( TANGENTSPACE_NORMALMAP )
    varying vec3 vViewPosition;
#endif

#include <packing>
#include <uv_pars_fragment>
#include <normal_pars_fragment>
#include <bumpmap_pars_fragment>
#include <normalmap_pars_fragment>
#include <logdepthbuf_pars_fragment>
#include <clipping_planes_pars_fragment>

void main() {
    #include <clipping_planes_fragment>
    #include <logdepthbuf_fragment>
    #include <normal_fragment_begin>
    #include <normal_fragment_maps>

    gl_FragColor = vec4( normalize( normal ) * colorA + 0.5, 1.0 );

    #ifdef OPAQUE
        gl_FragColor.a = 1.0;
    #endif 
}
</script> How normal shaders work A normal shader calculates the color of a pixel by the calculation . As such, we can swap out the first for a custom color, that being our . We can then add both of these vertex and fragment shader to our Javascript like so: normalize( normal ) * 0.5 + 0.5 0.5 uniform colorA let sphereMesh = new THREE.ShaderMaterial({
    uniforms: {      
        colorA: {type: 'vec3', value: new THREE.Vector3(0.5, 0.5, 0.5)},

    },
    vertexShader: document.getElementById('vertex').textContent,
    fragmentShader: document.getElementById('fragment').textContent,
}); It's important when learning WebGL to know that this is how it works under the hood. Shaders are really important to do things in 3d - so it's good to know how to define them and manipulate them. Manipulating our Sphere's Geometry We can then create a three.js clock to track time. We use that time to create noise, using our imported noise function. Noise is just randomness that will help create the effect of a randomly morphing sphere. After that, three.js also provides a function for us to add animation keyframes, and we can use the aforementioned three.js clock to animate that sphere. let noise = openSimplexNoise.makeNoise4D(Date.now());
let clock = new THREE.Clock();

renderer.setAnimationLoop( () => {
    // Get the time
	let t = clock.getElapsedTime();
    sphereGeometry.positionData.forEach((p, idx) => {
        // Create noise for each point in our sphere
        let setNoise = noise(p.x, p.y, p.z, t * 1.05);
        // Using our Vector3 function, copy the point data, and multiply it by the noise
        // this looks confusing - but it's just multiplying noise by the position at each vertice
        v3.copy(p).addScaledVector(p, setNoise);
        // Update the positions
        sphereGeometry.attributes.position.setXYZ(idx, v3.x, v3.y, v3.z);
    })
    // Some housekeeping so that the sphere looks "right"
    sphereGeometry.computeVertexNormals();
    sphereGeometry.attributes.position.needsUpdate = true;
    // Render the sphere onto the page again.
	renderer.render(scene, camera);
}) Now our sphere will start morphing! I repeated this for the plane behind the sphere, too. I used a here, with just a basic mesh, that makes it look like a wireframe. The code for that bit, along with everything else, . BoxGeometry is available on CodePen Conclusion Making 3d shapes on the web is a great frontend skill to have. Although a lot can be done in CSS and HTML, ome effects can only be achieved through 3d, and three.js provides the perfect platform to do that on. I hope you've enjoyed this quick guide to creating a 3d morphing sphere in three.js and Javascript. If you'd like more Javascript content, . you can read all my other stuff here Also published here.

Create a Morphing 3D Sphere in Javascript with Three.js

This post analyzes ops & maintenance solutions with experiences taken from real production scenarios for data sharding and other functions provided by ShardingSphere-Proxy Version 5.1.0. Unless otherwise specified, in the following examples, when we say “database” we refer to MySQL. What does ShardingSphere-Proxy do? ShardingSphere-Proxy allows users to use Apache ShardingSphere just as if it were a native database. To gain a better understanding of what’s ShardingSphere-proxy, let’s take a look at its definition provided, by : Apache ShardingSphere’s official website ShardingSphere-Proxy is a transparent database proxy that provide a database server containing database binary protocols designed to support heterogeneous languages. Currently, it supports MySQL and (and PostgreSQL-based databases, such as ) and any related terminals (such as , Workbench, etc.) that are compatible with MySQL or PostgreSQL protocols to operate data. It’s a DBA-friendly tool. PostgreSQL openGauss MySQL Command Client It’s worth noting that ShardingSphere-Proxy is a service process. In terms of client-side program connections, it is similar to a MySQL database. Why you need ShardingSphere-Proxy ShardingSphere-Proxy is a good choice when: sharding rules or other rules are used; because data will be distributed across multiple database instances, inevitably making management inconvenient. non-Java developers need to leverage ShardingSphere capabilities. 1. Application scenarios There are many scenarios where ShardingSphere-JDBC is used for data sharding. If you have a user table and need to perform horizontal scaling with Hash for the User ID property, the way the client connects to the database is like this: Below are three real production scenarios: A testing engineer wants to see the information of user ID 123456 in databases & tables, and you need to tell the engineer which subtable the user is in. You need to find out the total user growth in 2022 and overall user information for drafting a yearly report. Your company is going to hold its 8th-anniversary event and you are required to provide a list of active users who have been registered for over 8 years. Since the data is distributed across database shards and table shards, it is not easy to complete the above-mentioned tasks. If you develop code every time to satisfy these temporary requirements, it’ll be inefficient, to say the least. ShardingSphere-Proxy is perfect for these scenarios. ShardingSphere-Proxy hides the actual backend databases, so the user operates the client-side the same way as a database. For example, is split into several real tables at the database level, that is from to . While operating ShardingSphere-Proxy on the client-side, the user only needs to know one logical table , and routing to the real tables is executed inside ShardingSphere-Proxy. t_user t_user_0 t_user_9 t_user The logical name of the horizontally-scaled databases/tables with the same structure. A logical table is the logical identifier of tables in SQL. For example, user data is sharded into 10 tables according to the significant digits of the primary key, that is, to , and their common logical table is named . 1. Logical table: t_user_0 t_user_9 t_user The physical table actually exists in databases after scale-out, that is, the above-mentioned to . 2. Actual table: t_user_0 t_user_9 2. The differences between ShardingSphere-JDBC and ShardingSphere-Proxy After reading the above description, you probably feel that ShardingSphere-Proxy and ShardingSphere-JDBC are so similar. So what are the differences between the two? Check out more on the differences between the two below: ShardingSphere-JDBC is a package. Its bottom layer completes SQL parsing, routing, rewriting, execution, and other processes by rewriting JDBC components. You should add the configuration files to implement the corresponding functions in the project, making it intrusive to applications. .jar ShardingSphere-Proxy is a process service. In most cases, it is positioned as a productivity tool to assist operations. It disguises itself as a database, making itself non-intrusive to applications. The SQL execution logic in ShardingSphere-Proxy is the same as in ShardingSphere-JDBC because they share the same kernel. Since ShardingSphere-Proxy is non-intrusive to applications, and it shares the same kernel with ShardingSphere-JDBC — so why do we still need ShardingSphere-JDBC? When an application directly operates databases through ShardingSphere-JDBC, there is only one network I/O. However, when the application connects to ShardingSphere-Proxy, one network I/O, and then ShardingSphere-Proxy operates databases, and another network I/O occurs, in total two network I/O requests. There is one more layer of an application called link, which is more likely to cause a data traffic bottleneck and potential risks to the application. In general, it’s suggested that an application should be used together with ShardingSphere-JDBC. Of course, ShardingSphere-JDBC and ShardingSphere-Proxy can be deployed simultaneously with a hybrid architecture. ShardingSphere-JDBC is suitable for high-performance lightweight Online Transaction Processing (OLTP) applications developed in Java, while ShardingSphere-Proxy is perfect for Online Analytical Processing (OLAP) applications and scenarios for managing and operating sharding databases. Quick Start Guide There are three setup methods to install ShardingSphere-Proxy: binary package, Docker, and Helm. Stand-alone deployment and clustered deployment are also provided. Here, we take the standalone binary package as an example: Get the ShardingSphere-Proxy binary installation package ; at this link Decompress it and then modify and files starting with the prefix to configure sharding, read/write splitting and other functions; conf/server.yaml config- If you use Linux as operating system, please run For Windows operating systems, please run to bootup ShardingSphere-Proxy. bin/start.sh. bin/start.bat The file directory looks like this: ├── LICENSE
├── NOTICE
├── README.txt
0009 ─ ─ bin #Start/stop script
0009 ─ ─ conf #service configuration, data sharding, read/write splitting, data encryption, and other function configuration files
├── lib # Jar package
└── licenses Download the ( ) or ( ) into the package. 1. Copy MySQL Java connector to the ext-lib package driver mysql-connector-java-5.1.47.jar Click the link to download mysql-connector-java-8.0.11.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar ext-lib Note: There is no package in the initial directory, so you need to create one yourself. ext-lib For the default operation mode is . Below is how to configure the standalone operation mode. 2. Modify the conf/server.yaml configuration file server.yaml Cluster Mode mode:
Type: Standalone #Standalone mode
repository:
type: File
props:
Path: /Users/xxx/software/apache-shardingsphere-5.1.0-shardingsphere-proxy/file #persistent file paths such as metadata configuration
Overwrite: false #Overwrite the existing metadata？
Rules: #Verification Info
- !AUTHORITY
Users: #Init user
- root@%:root
- sharding@:sharding
provider:
type: ALL_PRIVILEGES_PERMITTED
- !TRANSACTION
defaultType: XA
provider Type: Atomikos
- !SQL_PARSER
sqlCommentParseEnabled: true
sqlStatementCache:
initialCapacity: 2000
maximum Size: 65535
concurrencyLevel: 4
parseTreeCache:
initialCapacity: 128
maximum Size: 1024
concurrencyLevel: 4
Props: #public configuration
max-connections-size-per-query: 1
kernel-executor-size: 16  # Infinite by default.
proxy-frontend-flush-threshold: 128  # The default value is 128.
proxy-opentracing-enabled: false
proxy-hint-enabled: false
sql-show: false
check-table-metadata-enabled: false
show-process-list-enabled: false
# Proxy backend query fetch size. A larger value may increase the memory usage of ShardingSphere Proxy.
# The default value is -1, which means set the minimum value for different JDBC drivers.
proxy-backend-query-fetch-size: -1
check-duplicate-table-enabled: false
proxy-frontend-executor-size: 0 # Proxy frontend executor size. The default value is 0, which means let Netty decide.
# Available options of proxy backend executor suitable: OLAP(default), OLTP. The OLTP option may reduce time cost of writing packets to client, but it may increase the latency of SQL execution
# and block other clients if client connections are more than `proxy-frontend-executor-size`, especially executing slow SQL.
proxy-backend-executor-suitable: OLAP
proxy-frontend-max-connections: 0 # Less than or equal to 0 means no limitation.
sql-federation-enabled: false
# Available proxy backend driver type: JDBC (default), ExperimentalVertx
proxy-backend-driver-type: JDBC Note: if you start a standalone ShardingSphere-Proxy and later need to change Proxy configurations, you need to set to true. By doing so, ShardingSphere-Proxy will reload the metadata after startup. mode.overwrite Execute the bootup command: . The default port , and the port can be replaced by adding parameters to the start script command: . 3. Start ShardingSphere-Proxy sh bin/start.sh is 3307 sh bin/start.sh 3308 To check whether ShardingSphere-Proxy started successfully, execute the check log command: . tail -100f logs/stdout.log The following information placed on the last line means that the startup is successful: [INFO ] xxx-xx-xx xx:xx:xx.xxx [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy Standalone mode started successfully Scenarios and applications Based on actual production scenarios, we’d like to show you how you can utilize ShardingSphere-Proxy to meet your expectations. 1. Initialize the database & table # CREATE DATABASE
CREATE DATABASE user_sharding_0;
CREATE DATABASE user_sharding_1;
# CREATE TABLE
use user_sharding_0;
CREATE TABLE `t_user_0` (
`id` bigint (20) NOT NULL,
`user_id` bigint (20) NOT NULL,
`create_date` datetime DEFAULT NULL,
PRIMARY KEY (`id`)) ENGINE = InnoDB DEFAULT CHARSET = latin1;
CREATE TABLE `t_user_1` (
`id` bigint (20) NOT NULL,
`user_id` bigint (20) NOT NULL,
`create_date` datetime DEFAULT NULL,
PRIMARY KEY (`id`)) ENGINE = InnoDB DEFAULT CHARSET = latin1;
use user_sharding_1;
CREATE TABLE `t_user_0` (
`id` bigint (20) NOT NULL,
`user_id` bigint (20) NOT NULL,
`create_date` datetime DEFAULT NULL,
PRIMARY KEY (`id`)) ENGINE = InnoDB DEFAULT CHARSET = latin1;
CREATE TABLE `t_user_1` (
`id` bigint (20) NOT NULL,
`user_id` bigint (20) NOT NULL,
`create_date` datetime DEFAULT NULL,
PRIMARY KEY (`id`)) ENGINE = InnoDB DEFAULT CHARSET = latin1; 2. Initialize sharding configuration in Proxy schemaName: sharding_db
dataSources:
ds_0:
url: jdbc:mysql://127.0.0.1:3306/user_sharding_0?serverTimezone=UTC&useSSL=false
username: root
password: root
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_1:
url: jdbc:mysql://127.0.0.1:3306/user_sharding_1?serverTimezone=UTC&useSSL=false
username: root
password: root
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
rules:
- !SHARDING
tables:
t_user:
actualDataNodes: ds_${0..1}.t_user_${0..1}
tableStrategy:
standard:
shardingColumn: user_id
shardingAlgorithmName: t_user_inline
keyGenerateStrategy:
column: user_id
keyGeneratorName: snowflake
bindingTables:
- t_user
defaultDatabaseStrategy:
standard:
shardingColumn: user_id
shardingAlgorithmName: database_inline
defaultTableStrategy:
none:
shardingAlgorithms:
database_inline:
type: INLINE
props:
algorithm-expression: ds_${user_id % 2}
t_user_inline:
type: INLINE
props:
algorithm-expression: t_user_${user_id % 2}
KeyGenerators:
snowflake:
type: SNOWFLAKE 3. Test sharding configuration Use the MySQL terminal command to connect to the ShardingSphere-Proxy server. If you deploy databases in Docker containers, you need to add native . Because accessing 127.0.0.1 in the container is blocked. -h ip #replace {xx} with actual parameter
mysql -h {ip} -u {username} -p{password} -P 3307
#example command
mysql -h 127.0.0.1 -u root -proot -P 3307 ShardingSphere-Proxy supports , , , , and other database management connectors. Navicat MySQL DataGrip WorkBench TablePlus After confirming the connection is successful, query the databases and make sure they are consistent with those in the configuration file. mysql> show databases;
+-------------+
| schema_name |
+-------------+
| sharding_db |
+-------------+ 1 row in set (0.02 sec) Execute the new statement, insert 6 pieces of user data (3 pieces for 2021, and 3 pieces for 2022). t_user mysql> use sharding_db;
mysql> INSERT INTO t_user (id, user_id, create_date) values(1, 1, '2021-01-01 00:00:00'), (2, 2, '2021-01-01 00:00:00'), (3, 3, '2021-01-01 00:00:00'), (4, 4, '2022-01-01 00:00:00'), (5, 5, '2022-02-01 00:00:00'), (6, 6, '2022-03-01 00:00:00');
Query OK, 6 rows affected (0.16 sec)
mysql> select * from t_user;
+----+---------+---------------------+
| id | user_id | create_date         |
+----+---------+---------------------+
|  2 |       2 | 2021-01-01 00:00:00 |
|  4 |       4 | 2022-01-01 00:00:00 |
|  6 |       6 | 2022-03-01 00:00:00 |
|  1 |       1 | 2021-01-01 00:00:00 |
|  3 |       3 | 2021-01-01 00:00:00 |
|  5 |       5 | 2022-02-01 00:00:00 |
+----+---------+---------------------+ At this time, the data is in the and databases respectively. user_sharding_0 user_sharding_1 Scenario 1: How to locate data information? Since ShardingSphere-Proxy has logically aggregated the tables, you can query them directly. mysql> select * from t_user where user_id = 1;
+----+---------+---------------------+
| id | user_id | create_date         |
+----+---------+---------------------+
|  1 |       1 | 2021-01-01 00:00:00 |
+----+---------+---------------------+
1 row in set (0.01 sec) Scenario 2: How to check user growth in 2022 and user information? mysql> select count(*) from t_user where create_date > '2022-00-00 00:00:00';
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.10 sec)
mysql> select * from t_user where create_date > '2022-00-00 00:00:00';
+----+---------+---------------------+
| id | user_id | create_date         |
+----+---------+---------------------+
|  4 |       4 | 2022-01-01 00:00:00 |
|  6 |       6 | 2022-01-01 00:00:00 |
|  5 |       5 | 2022-01-01 00:00:00 |
+----+---------+---------------------+
3 rows in set (0.02 sec) Scenario 3: How to get a list of active users who have been registered for over 8 years? Refer to the above code and you will know how to cope with it. Conclusion This post summarizes the basic concepts of ShardingSphere-Proxy based on the actual data sharding production scenarios and demonstrates how ShardingSphere-Proxy faces these scenarios. After reading this article, you should be able to: Understand why ShardingSphere-Proxy is an excellent product to assist developers in DevOps. Know the differences between ShardingSphere-JDBC and ShardingSphere-Proxy, their advantages and disadvantages, and how they are implemented. Since you now have a better understanding of ShardingSphere-Proxy, we believe, it will be easier for you to study its source code. To know more about ShardingSphere, please visit or our community’s previous blogs such as Create a Distributed Database Solution Based on PostgreSQL/openGauss. Apache ShardingSphere's official website Also published . here

A Quick Start Guide to Use ShardingSphere-Proxy in Real Production Scenarios 

The Cost of Building 5G Infrastructure Have you ever wondered how much it will cost to build the 5G infrastructure? According to GSMA Intelligence’s CAPEX forecast, 5G will cost almost US $1 trillion to deploy over the next half-decade. With 5G, it will require more network densification, which means a lot of the capex investment will go into the RAN* (Radio Access Network.) In fact, the share of RAN in total capex will grow from 62 per cent in 2018 to 86 per cent in 2025 (please ). see figure 1 *In simple terms, a Radio Access Network (RAN) is part of a telecommunications system that connects individual devices such as our cell phones to other parts of a network through radio connections. The network operators are paying for the cost of building the 5G infrastructure to provide service to their customers and it has a huge impact to their revenue and market capitalization. The network operators have no choice but to find ways to lower their capex. Why Open RAN (O-RAN)? Since the share of RAN in capex is over 60% and rising, the most natural question to ask is how to make it cheaper and cost less to operate. And at the same time, there is increased pressure for the operators to complete their 5G rollouts. To deal with the challenge associated with upgrading and rolling out networks, several operators (AT&T, China Mobile, Deutsche Telekom, NTT DOCOMO and Orange) created the in February 2018 to drive the development of standards and technologies that would lead to Open RAN implementation. What they need is faster innovation, easier deployment, simplified operations, and a more competitive vendor ecosystem. O-RAN Alliance Open RAN will meet these needs by providing greater choice, flexibility and agility for network operators racing to deploy 5G. Open RAN also helps operators to speed service introduction and innovation, improve supply chain security, and reduce Opex and Capex. The O-RAN Alliance now has over 300 mobile operators, vendors, and research & academic institutions operating in the Radio Access Network (RAN) industry worldwide. Please see figures 2 & 3 above. You’re Crazy If You Don’t Embrace the Open RAN Ecosystem Recently, Dell’Oro Group raised its Open RAN revenue forecast by 50% as the strong momentum behind the technology has continued to grow. It now believes the cumulative open RAN revenue from 2020 to 2025 could soar to as high as $15 billion, with open RAN revenues accounting for more than 10% of the overall RAN market by 2025. The market is moving at a faster pace than expected, especially in the APAC region. With the US government providing more than $1.5 billion funding to support Open RAN and the Japanese government offering financial incentives and tax benefits for companies that develop, supply, and deploy Open RAN equipment, there is a growing concern of the future viability of a European ecosystem able to compete with other regions in the world. Feeling that Europe as a whole is lagging behind other regions such as the U.S. and Japan in developing Open RAN, Deutsche Telekom, Orange, Telecom Italia, Telefónica, and Vodafone are calling upon European Union member states, policymakers, and industry to embrace the Open RAN ecosystem with the new report — “ .” Building an Open RAN Ecosystem For Europe To understand why embracing Open RAN should be an easy decision, we need to have a better understanding of what is Open RAN and what are the benefits of Open RAN? What is Open RAN (O-RAN)? Open RAN is a set of open specifications defining interfaces between radio and base station RAN. The disaggregation of different RAN elements would provide new ways to mix and match network components by “opening up” the interfaces between them. It will drive down the costs of network deployments and allow new players to enter the RAN market. Please see figure 4 below. Future Radio Access Networks will be based on a wide range of solutions deployments, one size will no longer fit all**.** Continued 4G & 5G traffic growth also requires a significant increase in RAN capacity and network operators to maximize existing investments. Open RAN enables all types of RAN deployment models: distributed, centralized, virtualized, macro & small cell. And it is built on general-purpose computing hardware with carrier-grade reliability. Along with resource pooling, load balancing, and spectral efficiency, Open RAN enables network operators to maximize resource utilization and deliver new network edge services at a lower cost. Many Paths to Success According to Dell’Oro Group, the mix between incumbent RAN players and new challengers with smaller shares is “relatively healthy at this juncture, with smaller RAN suppliers including Airspan, Altiostar, Fujitsu, Mavenir, NEC, and Parallel Wireless accounting for the majority of the Open RAN revenues in 1H21.” The beauty of Open RAN is that network operators can mix and match hardware with different suppliers. For example, the O-RAN DU reference designs from Supermicro can support RAN software from either Mavenir or Parallel Wireless. And thanks to Moore’s Law, each generation of processors is getting increasingly powerful, and the ever-falling cost of processing power is making Open RAN even more appealing economically. Please see Figure 5 above. Conclusion Open RAN marks a profound change in how the network operators is building the wireless infrastructure. This change is driven by an incredibly powerful value proposition, driving efficiencies both in operations and economics. The Open RAN specifications also help cultivate innovation as resources are freed up to allow the network operators to focus on new products and growth. While it’s still ramping, Open RAN will help the network operators to cut billions of dollars in capital spending and speed up the rollout of the 5G network. Wouldn’t it be crazy if we don’t embrace the Open RAN ecosystem? References Intelligence Brief: How much will 5G cost? The year is 2019 and 5G is finally here. Or is it? At MWC19 Barcelona, there were so many 5G announcements that it… www.mobileworldlive.com Dell'Oro bumps revenue forecast 50% as open RAN gains steam Dell'Oro Group raised its open RAN revenue forecast by 50%, citing strong momentum behind the technology in the first… www.fiercewireless.com European Telcos Sound Alarm Over Flagging Open RAN Progress European telecom companies recently sounded an alarm that they may be falling behind the rest of the world in their… spectrum.ieee.org 5G Testing: What Is O-RAN? - Part 1 Open radio access network (O-RAN) is an operator-led alliance that defines the next-generation radio access network… blogs.keysight.com

Edge Computing is So Fun Part 6: Why You Need to Embrace The Open RAN Ecosystem

22 Alternatives to Trello for Simple Project Management

32 Project Management Tools: An Ultimate, Unbiased List

29 Asana Alternatives: The Ultimate Breakdown

Top 33 Jira Alternatives For Agile Project Management Teams

This is a thing that you can do to clean up your local git repo AFTER doing a commit and before pushing to origin. PREWORK 1. Download BFG.jar https://rtyley.github.io/bfg-repo-cleaner/   https://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar 2. Create directory structure mkdir - ~/.binaries/ p 3. Move bfg in to place and chmod it mv bfg .jar ~/.binaries/ && chmod ~/.binaries/bfg .jar -1.13 .0 777 -1.13 .0 4. setup BASH_PROFILE ' bfg= ' >> ~ >> ~ ' bfg= ' >> ~ >> ~ echo alias "java -jar ~/.binaries/bfg-1.13.0.jar" /.bash_profile echo "export PATH=/Users/$(whoami)/.binaries:$PATH" /.bash_profile echo alias "java -jar ~/.binaries/bfg-1.13.0.jar" /.bash_profile echo "export PATH=/Users/$(whoami)/.binaries:$PATH" /.bash_profile 5. source it ~/.bash_profile source Cleaning repo with BFG Now what do we do. I will create a new folder, and then create a git repo in that folder. This would be the same as doing a git clone... mkdir testing-bfg cd testing-bfg/ git init Now I am going to make a file and add some things to it vi README.md PASSWORD=my_password
AWS_TOKEN=1234 # do things # do things that are bad add it to my git repo git add README.md make a commit to the repo for the file git commit - -m a "I did an initial commit and it is bad" I am going to attempt to push to master, or don't when I realize I did a bad thing origin master
*** pre-receive hook fires stops me from so the fails. git push and pushing, push OR I don't push, but the commit is already in my history, I need to clean it. This is how you fix it. In order for BFG to run properly you cannot run it from the source of your git repo, you need to be at least out by one folder. cd ../
vi ~/my-mess AWS_TOKEN=X1V34
PASSWORD= #this file is now located at your home folder as denoted by the ~/   thingy #my-mess file # these are the strings that you want out of your repo and commit history Now we have our dirty git repo located at . testing-bfg We also have a text file with the "things we want to remove from the repo" that is located at ~/my-mess We are going to run the bfg command which is aliased to but you could also do this without the alias, bfg java -jar bfg-1.13.0.jar The is telling bfg to rewrite my current history in all my branches and make clean. --no-blob-protection bfg ~ testing-bfg/ --replace-text /my-mess --no-blob-protection Now you will have your file in a "modified" state according to git. This is because your local file still has the BAD words in it, however your commit history does not. Therefore it is considered modified. Now, go through and change your local file to fix your shame and do a new commit and move forward. PROFIT

How To Cleanup Your Local Git Repository Using BFG

24 Experts Weigh In: How Do You Get Traffic Without Budget?

The practical guide for Building REST API in Nodejs and MongoDB include Passport and JWT

Swift // Defining a Base Classclass Vehicle {var currentSpeed = 0.0var description: String {return "traveling at \(currentSpeed) miles per hour"}func makeNoise() {// do nothing - an arbitrary vehicle doesn't necessarily make a noise}}let someVehicle = Vehicle()print("Vehicle: \(someVehicle.description)")// Vehicle: traveling at 0.0 miles per hour // Subclassingclass SomeSubclass: SomeSuperclass {// subclass definition goes here}class Bicycle: Vehicle {var hasBasket = false}let bicycle = Bicycle()bicycle.hasBasket = true bicycle.currentSpeed = 15.0print("Bicycle: \(bicycle.description)")// Bicycle: traveling at 15.0 miles per hour // Subclasses can themselves be subclassedclass Tandem: Bicycle {var currentNumberOfPassengers = 0}let tandem = Tandem()tandem.hasBasket = truetandem.currentNumberOfPassengers = 2tandem.currentSpeed = 22.0print("Tandem: \(tandem.description)")// Tandem: traveling at 22.0 miles per hour // Overridingclass Train: Vehicle {override func makeNoise() {print("Choo Choo")}}let train = Train()train.makeNoise()// Prints "Choo Choo" // Overriding Property Getters and Settersclass Car: Vehicle {var gear = 1override var description: String {return super.description + " in gear \(gear)"}}let car = Car()car.currentSpeed = 25.0car.gear = 3print("Car: \(car.description)")// Car: traveling at 25.0 miles per hour in gear 3 JavaScript // Defining a Base Classclass Vehicle {constructor() {this.currentSpeed = 0.0}get description() {return `traveling at ${currentSpeed} miles per hour`}function makeNoise() {// do nothing - an arbitrary vehicle doesn't necessarily make a noise}}let someVehicle = Vehicle()console.log(`Vehicle: ${someVehicle.description}`)// Vehicle: traveling at 0.0 miles per hour // Subclassingclass SomeSubclass extends SomeSuperclass {contructor() {super();}// subclass definition goes here}class Bicycle extends Vehicle {contructor() {super();this.hasBasket = false}}let bicycle = Bicycle()bicycle.hasBasket = true bicycle.currentSpeed = 15.0console.log(`Bicycle: ${bicycle.description}`)// Bicycle: traveling at 15.0 miles per hour // Subclasses can themselves be subclassedclass Tandem extends Bicycle {contructor() {super();this.currentNumberOfPassengers = 0}}let tandem = Tandem()tandem.hasBasket = truetandem.currentNumberOfPassengers = 2tandem.currentSpeed = 22.0console.log("Tandem: \(tandem.description)")// Tandem: traveling at 22.0 miles per hour // Overridingclass Train extends Vehicle {contructor() {super();this.currentNumberOfPassengers = 0}function makeNoise() {console.log("Choo Choo")}}let train = Train()train.makeNoise()// Prints "Choo Choo" // Overriding Property Getters and Settersclass Car extends Vehicle {contructor() {super();this.gear = 1}get description() {return `${super.description} in gear ${gear}`}}let car = Car()car.currentSpeed = 25.0car.gear = 3console.log(`Car: ${car.description}`)// Car: traveling at 25.0 miles per hour in gear 3 More of Swift and JavaScript comparison snippets

Swift and JavaScript comparison snippets(9) — Inheritance

In the summer of 1976, Canada was hosting its first ever Olympics in Montreal. On the west coast, a child was born who would develop a larger than life personality. Although an excellent marathoner, Little Stevie Black as he became known, never quite made it to the Games. His notoriety is for something else, asking questions. Little Stevie Black now nearly six feet tall and has the dark complexion typical of someone who’s just returned from a Mexican holiday. He’s highly presentable with a smidge of beard stubble and a good-humoured disposition. Much like filmmaker Louis Theroux, he has the uncanny ability to make he talks to feel comfortable within moments. anyone With determined inquisitiveness, Little Stevie Black discovers things about you that you would rarely grant others access to. That’s because when you first meet him, he makes you feel you’ve known him all your life. And it’s part and parcel of what makes him the quintessential maven. The word derives from the Yiddish word , which literally translates as ‘one who understands’. True mavens, like Little Stevie Black, yearn to transmit their knowledge to others. Algorithms, on the other hand, are a set of rules that are followed in order to solve problems. The ‘understanding’ is achieved through . With sets of data, algorithms, as we know, can grow exponentially smarter and more powerful over time. maven myvn inputs A Curious Mind An exceptionally curious child, it’s no wonder that Little Stevie Black didn’t drive his parents right off the deep end. With the patience of saints, they would unwaveringly humour their son. As Sir Ken Robinson surmises they knew that: “Curiosity is the engine of achievement.” Instead of squelching their child’s deep internal motivations to be inquisitive, his parents encouraged him to question things — to delve deeper into issues. This thirst for knowledge enabled him to better understand how his world worked. But as Little Stevie Black grew up and his questions became more sophisticated, he was forced to wizen up, rely less on his parents and seek the answers himself. He became increasingly better at exercising sound judgment in a myriad of situations. It’s not surprising he’s now a seasoned Crown Counsel with the British Columbia Prosecution Service. Curiosity is one key to being a maven, but there is also the willingness to transmit one’s findings. In today’s information-heavy world, it’s all too commonplace to rely on algorithms to relay the information we want — or at least think we want. With Google of all searches online it’s easy to see where things might go astray. No matter how innovative the algorithms of the search engine are, they still produce the results determine. Google, and their zealots across the web have tried to replace Little Stevie Black’s mavenhood with super slick algorithms. accounting for three-quarters they Scaling Personality With his giving spirit, Little Stevie Black shares the wealth of knowledge he accumulates. He was the first to tell me about Nike Pegasus Runners, Breaking Bad, ClassPass, Balinese life — not to mention some of the finest Gelato joints in town and tastiest restaurants on earth. Little Stevie Black married another maven, of course. And now they are double trouble. His wife Nikki, an ex-New Yorker and bonafide maven herself, is a powerful driving force behind their insider knowledge. Nowadays her husband may very well find himself in the backseat and riding on her coattails. Whichever way, together they offer some serious mavenhood to the neighbourhood. Yet unlike algorithms, the duo is limited to a finite network. There are only so many people in their social circle they can bless with unique insights they garner from their inquiries. Sure Amazon, Spotify, and Netflix algorithms do a pretty darn good job for you and indeed they get increasingly better as we use them. But they do lack one very important quality: personality. Care and Context The sad reality is that mavens like Little Stevie Black and Nikki are not being heard, or for that matter are rarely consulted. Maven wannabees have taken to Instagram to amplify their recommendations or carve out a living. In the process, they dilute their mavenhood if not abandon it altogether. Sure, I’ve dreamt about the Little Stevie Black maven effect. But it’s precisely because scaling personality is an extremely delicate dance that so often falls short. Platform technologies like OpenTable or TripAdvisor do an OK job, but you still don’t trust them wholeheartedly as you would a maven. scaling Without cross-referencing your online results with a human who actually cares about your experience, within the given context of your unique situation, as well as granting you the luxury to cultivate your own curiosity — you often miss out on discovering that truly special something. Know-it-Somes Algorithms seek to know everything — in effect to become . Mavens, on the other hand, are always inquiring, always deepening and advancing their particular tastes. Even when they have found something delightful on one corner, they can’t wait to discover what may lie around the next. Mavens are smart enough to know they can’t and shouldn’t try to know it all. If they did, there wouldn’t be any more restaurants to discover and tell their friends about. They are content being know-it-alls know-it-somes. An algorithm might be great for helping you find what you’re looking for, and admittedly, pretty good at predicting what you didn’t even know you were looking for. But it will never be as helpful as your maven bestie, who more often than not, knows you better than you know yourself. ∆

Why Algorithms Will Never Be Mavens

How product managers at top companies control releases and accelerate the product lifecycle thebridgesummit.org The Product Manager You are the king or queen of the product. You’re the engineer, the designer, the business mind, the marketer… in fact, you are a little bit of everything. You are the product owner, but more importantly, you are the leader. In his article, Prasad Thammineni (CPO of Choose Energy) outlines the : 9 qualities of a great product manager Understanding the current needs of the product and how it fits with the company’s overall vision. Strategic Thinker — Love what you make, love what you do. Passion for Products — and understand the customer’s wants and needs. Empathizes with the Customer — Gather insights from your customers, learn their pain points and discover opportunities. Interviews Customers — Understands that user experience is paramount for product success. Aspires to Build Great Experiences — Identify the key metrics and track performance, establish baselines to measure success. Keeps Score — Keep things in order, understand what needs to be completed first and what needs to wait. Use customer feedback for prioritization. Ability to Prioritize — Management is a collaborative process, you cannot be dictatorial and expect to inspire your team. Collaborative Leader — Action-oriented, you get things done. Execution — To embody these 9 principles, you’ll typically see PMs use a myriad of tools to organize teams, manage development, QA, and track benchmarks (JIRA, Aha, Mavenlink, and about 10,000 others). Every aspect of effective product management seems to come with its own suite of tools, but two areas are generally lacking: and , which I’ll explore in greater depth. Customer Empathy Customer Feedback (interviews, testing) Customer Empathy As a Product Manager, you must empathize with the needs and wants of your target audience. As (PM at FullContact) writes: Catherine Shyu “The product manager is responsible for knowing the customer like the back of their hand — their frustrations, their daily thoughts, the context in which they use the product, and their needs.” It’s one thing to ask your customers for feedback, to interview them during development, and to have them test your product before it’s launched. It’s another thing to have your customers test your product in a real environment or to provide certain customers with a personalized experience. A key to empathy is the ability and capacity to take action on feedback. As notes, Jennifer Winter “Collaborative and creative empathy doesn’t equal sympathy. The goal of empathy is to support and understand the intent and purposes of others, which in turn will help you obtain insightful feedback.” Genuine Feedback User interviews, in-person testing, assigning tasks, and conducting basic user research are fundamental to developing a successful user experience. However, they each have a common limitation: artificial feedback. When we sit down with a user and ask he or she to do something, we have taken that user out of their natural environment by forcing them to unnaturally provide feedback. The feedback we receive is still valuable, but it’s not purely genuine. comes from testing real users in a real environment without them knowing they are being tested. Imagine being able to test a new feature or experience on 1% of your actual users before launching to the other 99%. Genuine feedback Now, imagine if you could just roll back Feature A completely and then use interviews/testing to figure out why the feature wasn’t being used, and then roll it out again. For example, I could release to 1% of my users. If they never use the feature or only use it once, then that feedback is deafening. Feature A enables this genuine feedback loop. Feature flag driven development Feature Flags and Genuine User Feedback PMs at Google, Facebook, and Amazon have . This allows them to gather user feedback on a feature before it has been fully released. It also helps them be more responsive to feedback without having to leave a “bad feature” out on the market for too long. integrated feature flags into their development cycle Let’s say Facebook wants to release a new timeline feature called . This feature reorders timeline stories in order to increase user engagement. Let’s also assume that has gone through all the rigors of internal testing, user interviews, and whole gamut of proper UX research. Timeline Beta Timeline Beta For launch, Facebook wraps in a feature flag, allowing Facebook to have complete control over who sees the feature. Timeline Beta On day 1, they roll out the feature to 0.5% of Facebook users to see how they like it. On day 2, they see a 20% decline in user engagement for those users who had . Timeline Beta So, on day 3, Facebook rolls back via the feature flag and begins the cycle anew. Timeline Beta Imagine if Facebook released to all of its users without an easy mechanism to revert to the previous version. They could have seen millions in lost revenue, caused a social media uproar, and turned users away. This is the power of feature flag driven development: the ability to take full control over feature releases and test features in a real world environment. Timeline Beta Getting Started With Better Product Management Product managers have enormous responsibility. They must coordinate multiple teams, identify business objectives, manage timelines, and take ultimate responsibility for the success of a release. Luckily, integrating feature flags into a development cycle does not require a complete process transformation or months of retraining. If you’re already practicing continuous delivery, feature flagging can become an integral part of your dev cycle in no time at all. If you’re an agile team, then you can think of feature flagging as a way to iterate safely and intelligently, without the risk of crippling your application or upsetting all of your users. This is what allows product managers at top companies (Facebook, Google, Amazon) to safely manage the product lifecycle and gather genuine user feedback. — Thanks for reading! I love writing for you, so claps and follows are greatly appreciated :)

Silicon Valley’s Top Product Managers Use Feature Flags — And You Should Too

Before we begin, let’s be clear on terminology. When I refer to “working with data” in the context of I could mean one of two things: software development working with data, with perhaps Jupyter (née IPython) or the live interpreter Interactively and programs that primarily manipulate data Writing, testing, reading, reviewing maintaining In short: Python is awesome for interactive data analysis but terrible for writing long-lived programs dealing with complicated data structures. The second definition is perhaps overly broad, but I’ll clarify in a minute. Before that, let me be the first to say that is an language for interactively working with, or , data. The ecosystem of third-party packages and tools that have sprung up around data manipulation, visualization, and data science in general has been nothing short of remarkable. Python incredible exploring If working with interactive data is your nail, Python should be your hammer. But what about that second interpretation? Actually, it can be thought of as a logical extension of the first. Imagine you’re writing a program to query a database for a search term, do some sentiment analysis, and return the results in JSON. Working interactively with the database results, the results returned by your sentiment analysis library, and the JSON you produce is the natural first step. You’re still in “exploration mode”. Not really writing the program yet, just seeing what the data looks like and how you’ll need to manipulate it. Once you get a “feel” for the “shape” of the data at each step, you can begin to write your program. You’ll likely refer back to examples of the output you created during exploration when implementing the logic of your program. Particularly with deeply nested data structures (I’m looking at you, “everyone’s abuse of JSON…”), it’s often too difficult to keep the “shape” of the data at each stage in your head. But Python makes working with data , so your program is finished in no time. It works, it’s well-documented, and even has 100% test coverage. If you never need to return to this code, huzzah! Your job is done. easy Dynamic Typing Is The Root Of All Evil (j/k…kind of…) The very property of Python that made your program so easy to write is the same one that will make it difficult to review, read, and (most importantly) . Python’s dynamic type system means that, in most cases, you don’t have to enumerate the complete set of fields, types, and value constraints that define the data as it moves through your system. You can just jam it all in a ! Heterogeneous values FTW! maintain dict The task above would be much more laborious and time-consuming in a statically typed language like C or Go. In Go, for example, to parse and return a JSON response from some web API, you first need to create a whose fields and field-types match the structure of the response. Here is how one must prepare to work with a JSON response from (taken from their client library): struct exactly [etcd](https://github.com/coreos/etcd) type Response struct {// Action is the name of the operation that occurred. Possible values// include get, set, delete, update, create, compareAndSwap,// compareAndDelete and expire.Action string `json:"action"` // Node represents the state of the relevant etcd Node.Node *Node `json:"node"` // PrevNode represents the previous state of the Node. PrevNode is non-nil// only if the Node existed before the action occurred and the action// caused a change to the Node.PrevNode *Node `json:"prevNode"` // Index holds the cluster-level index at the time the Response was generated.// This index is not tied to the Node(s) contained in this Response.Index uint64 `json:"-"`} type Node struct {// Key represents the unique location of this Node (e.g. "/foo/bar").Key string `json:"key"` // Dir reports whether node describes a directory.Dir bool `json:"dir,omitempty"` // Value is the current data stored on this Node. If this Node// is a directory, Value will be empty.Value string `json:"value"` // Nodes holds the children of this Node, only if this Node is a directory.// This slice of will be arbitrarily deep (children, grandchildren, great-// grandchildren, etc.) if a recursive Get or Watch request were made.Nodes Nodes `json:"nodes"` // CreatedIndex is the etcd index at-which this Node was created.CreatedIndex uint64 `json:"createdIndex"` // ModifiedIndex is the etcd index at-which this Node was last modified.ModifiedIndex uint64 `json:"modifiedIndex"` // Expiration is the server side expiration time of the key.Expiration *time.Time `json:"expiration,omitempty"` // TTL is the time to live of the key in second.TTL int64 `json:"ttl,omitempty"`} The “ " part after each field describes what that field's name should be when the object is marshaled from a JSON message. And notice that, because contains a nested object ( ), we must fully define that nested object as well. json:... Response Node Note: to be fair, there are some shortcuts one might take in Go to reduce the need for a portion of the above, but they’re rarely taken (and for good reason). In Python, you’d be all like: result = make_etcd_call("some", "arguments", "here") If you wanted to see if the in question was a directory, you'd pound this out: node if result.json()['node']['dir']:# make magic happen... And the Python version is less code and takes less time to write than the Go version. “I Don’t See The Problem” The Python version is better, right? Let’s consider two definitions of “good code” so we can be clear what we mean by better. Code that is short, concise, and can be written quickly Code that is maintainable If we’re using the first definition, the Python version is “better”. If we’re using the second, The Go version, despite containing a boatload of boilerplate-ish definition code, . it’s far, far worse. makes clear the exact structure of the data we can expect in _result_ Boss: “What can you tell me about the Python version, just by looking at our code above?” Me: “Uh, it’s JSON and has a ‘node’ object which probably has a ‘dir’ field.” Boss: “What type of value is in ? Is it a boolean, a string, a nested object?” dir Me: "Uh, I dunno. It's truthy, though!" Boss: "So is everything else in Python. Is guaranteed to be part of the object in the response?" dir node Me: "Uh...." And I’ve met my “3-Uh” limit for describing what a portion of code does. If you refer to the Go version, you can answer those questions and sound like a damned genius in comparison. But these are . The answers to the questions in the Go version are self-evident. The answers for the Python version, not so much… exactly the sort of questions your peers should be asking in a code review Making Changes What happens when we need to make a change to the Python version? Perhaps we want to say “only if the directory was just created, not for every response with a directory?" make magic happen It’s pretty clear how to do that in the Go version. Compared to the Python version, the Go version is like the Library of Alexandria of s. For the Python version, We'll have to go look up the HTTP API documentation. Let's hope that: etcd Response we have nothing local to refer to in order to figure out the structure of _result_ and the change we need to make. etcd it exists it is well maintained the tubes aren’t clogged And this is a simple change we’re talking about on a simple JSON object. I could tell horror stories about what happens when you get knee-deep in Elasticsearch JSON responses… (spoiler alert: ). very very response['hits_']['hits_']['hits_']... The fun doesn't stop at just the code change, though. Remember, we're professionals, so all of our code is peer reviewed and unit-tested. After updating the code we can still barely reason about it. All of a sudden, we're back to that conversation between my boss and I where I say "Uh" a lot and he wonders why he didn't go into carpentry. making correctly Everybody Panic! I’ve painted a rather bleak picture of using Python to manipulate complex (and even not-so-complex) data structures in a maintainable way. In truth, however, it’s a shortcoming shared by dynamic languages. In the second half of this article, I’ll describe what various people/companies are doing about it, from simple things like the movement towards “live data in the editor” . In short, there’s a lot of interesting work going on in this space and of people are involved (notice the second presenter name in that Dropbox deck ). most all the way to the Dropboxian “type-annotate all the things” lot’s Originally published at jeffknupp.com on November 13, 2016. is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

How Python Makes Working With Data More Difficult in the Long Run

One thing I’ve found kind of strange since starting this internship is how old some of the software we use is. A lot of it is outdated and inefficient from a writers perspective. We can download apps from the Mac store for our own personal use but this only solves some problems. Many issues will exist until an agency wide shake up occurs. For compatibility and simplicities sake everyone in the firm should be running the same apps. And we should be power users always learning new features and shortcuts by running internal workshops. We could even release video tutorials on our blog. I guess there is some basic logic as to why we use a lot of the old slow tech. People like to stick with what they know and are afraid of change. Even if some kind of software is a million times better it would be difficult to get this type of person over the line. What I’d Do Now as an intern I obviously don’t have the power to change any of this. But here are some of my ideas as a start. We currently use Mavenlink. From my experience it does the job but lacks a really satisfying user experience. It feels forced and difficult to find content that you need. Last year I was constantly hearing about how good Slack is. Whilst doing a trial at a sports magazine I was invited to one of their Slack groups. To put it simply comparing Mavenlink to Slack is like comparing an iPhone to an old Nokia brick. Both get the job done but you get the picture. Slack could also solve another problem here which is time sheets. We currently have to use Windows Remote Desktop on Mac to get to Account Works which is glitchy at the best of times. It’s also been confusing to use as there is copies of the same company but with different codes. I’ve just done a quick search and looks like a perfect solution. Nikabot The biggest problem is the fact that we are slaves to Office 365. Outlook is horrible mail client that makes your eyes water when looking at it longer than 5 minutes. I can see the merits of email. It will never die but something has to give when it comes to its usage. The team should be hooked up to Airmail but we should also all be on . This would diminish the need to send internal mail. Messenger Microsoft Word isn’t that bad. I don’t mind using it but its nowhere near the best processor out there. Docs is by far and away the best for typing, editing, making suggestions and commenting on a collaborative project. Google tool Its integration with Google Drive is also incredibly easy to use for backing up files. Every project that I work on here, big or small, I’ll back it up on Drive and kind of use it as my portfolio. When I meet people I can show them my work on my phone no matter the file format. This also helps to explain what I actually do as a copywriter! My Workflow Ironically, I’m actually becoming less software based. I mostly draft and write all of my notes on paper. Firstly, I’ll do a brainstorm in flowchart style with a black sharpie. I’ll then use this as a reference to get a draft out on lined paper quickly. I like to write with double spaces as this leaves plenty of room for corrections. It’s also just a lot more readable. I then edit this draft in iA Writer which I believe is the best word processor in the app store. There is basically no formatting options which means you focus in on your writing distraction free. It’s extremely responsive when resizing which makes it easy to read something on Google Chrome and type. I also like its word count and reading time functionality. An extremely useful tool in hitting a brief. Note storage for specific projects is best kept in Evernote. This could even replace Slack in certain teams. I’m not using it a whole lot right now, but I certainly would be if I had an iPad Pro and an Apple pencil. For now I’ll just be getting my notes out on paper, even if things end up a little disorganized and unsearchable.

Stories, News & Mentions about MAVENOID

Mavenoid’s Stories on HackerNoon