1Flow

1flow

Authors:
(1) Anatol Guglielmi, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences;
(2) Boris Klain, Borok Geophysical Observatory of Schmidt Institute of Physics of the Earth, Russian Academy of Sciences;
(3) Alexey Zavyalov, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences;
(4) Oleg Zotov, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences and Borok Geophysical Observatory of Schmidt Institute of Physics of the Earth, Russian Academy of Sciences. Table of Links Abstract and Introduction Omori (1894) Hirano (1924) Discussion Conclusion and References Omori (1894) It is best to start describing the outstanding events of a century ago from afar. In 1850, John Milne was born in Liverpool, who in the future was to become a famous geophysicist, one of the founders of modern seismology [15]. He was educated in London, worked as a mining engineer in Newfoundland, as a geologist in the Sinai Peninsula, and from 1875 to 1895 worked in Tokyo at the invitation of the government of the Japanese Empire. It is curious that in search of adventure, Milne traveled to Tokyo for three months, mostly by land (via Siberia). In 1880, he created a horizontal pendulum seismograph - the first easy-to-use and sufficiently sensitive instrument for recording earthquakes. In 1887 Milne was elected a member of the Royal Society. He manages to convince the Society to allocate funds for the creation of a world network of seismic stations equipped with his instruments. Japan highly appreciated Milne's services to the country and the world. Emperor Meiji awarded him the Order of the Rising Sun and awarded him a lifetime pension of 1,000 yen. The University of Tokyo elected him honorary professor. Fusakichi Omori was a student of John Milne. He enjoyed the encouraging support of his teacher, as did all young Japanese seismologists of that time. On October 28, 1891, the earthquake with a magnitude of M = 8 occurred. Milne's seismographs recorded numerous aftershocks. Analysis of these aftershocks allowed Omori to formulate the law that bears his name [2]. It is worth mentioning that he was 26 years old at the time. Omori's Law states that after a strong earthquake, the frequency of aftershocks, i.e. tremors following the main shock, on average, decreases hyperbolically over time: Here k > 0, c > 0, t ≥ 0. This was the first law of earthquake physics, if we keep in mind the chronological sequence of outstanding discoveries in seismology. Intuition tells us that the Omori formula (6) has the characteristics of a fundamental law of nature. Let's explain what was said. Firstly, (6) can be rewritten in the equivalent form n kt = /, where k > 0, t > 0. In fact, contrary to popular belief, the parameter in (6) is free due to the homogeneity of the flow of time t. In other words, Omori’s law is one-parameter: the value of k generally characterizes the source of an earthquake over the entire history of the evolution of aftershocks. The general solution of equation (2) is Taking (1) into account, it is easy to see that Omori’s law (6) follows from (7) under the condition s = const . In this case k =1/σ , 0 c n =1/ σ . In other words, Omori's law is satisfied provided that the deactivation coefficient remains unchanged throughout the entire history of aftershock excitation. The found condition for the applicability of the Omori law is tough. Is it carried out experimentally? To answer the question, we compiled the Aftershock Atlas [8], which presents solutions to the inverse source problem. The characteristic size of the source was determined according to the method [16]. Analysis of solutions for several dozen events indicates that in all cases, at the first stage of the evolution of aftershocks, immediately after the main shock, the source deactivation coefficient remains unchanged within the limits of measurement accuracy. We call the time interval during which s = const the Epoch of Hyperbolic Evolution (EHE) , or Omori epoch. The duration of EHE varies from case to case from a few days to several months. Figure 1 shows a case of a fairly long EHE, almost equal to 100 days. There is a tendency for σ to decrease with increasing magnitude of the main shock [17]. The discovery of the Omori epoch is extremely significant. At the end of the epoch, the state of the source changes. Let's introduce parameter θ = dσ / dt. In the Omori epoch, θ = 0. The transition to a new state is indicated by a sharp jump in parameter θ (Figure 2). This suggests that the end of the Omori epoch is accompanied by a bifurcation of the earthquake source. A convincing illustration of this is Figure 3. It shows the result of processing and analyzing eight events. The transition from one mode of source deactivation to a qualitatively different mode is abrupt in the sense that its duration is much less than the duration of the Omori epoch. Let us summarize the preliminary results of studying the evolution of aftershocks by solving the inverse problem: 1The Omori formula (6) has the status of a law that is fulfilled with good accuracy in a real geophysical process. The applicability of Omori's law is strictly limited to the epoch of harmonic evolution of the earthquake source.


At the end of epoch of harmonic evolution, the source experiences a bifurcation. To conclude the section devoted to Fusakichi Omori, let us return to formulas (6) and (7). The law of aftershocks (7) formally differs from law (6) only in that in it time t is replaced by proper time τ . The concept of proper time and the corresponding term were introduced into physics only at the beginning of the 20th century. It would seem that Omori simply did not have the opportunity to use the concept of proper time in order to give formula (6) the flexibility and physical meaning that our formula (7) has. In this regard, we want to clarify that the source proper time (1) is not directly related to the relativistic proper time. Formulas of the form (1) express the uneven functioning of any dynamic system immersed in a nonstationary environment or located in nonstationary external force fields. In our case, formula (1) takes into account the non-stationary state of the source that experienced the main shock of the earthquake. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Anatol Guglielmi, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences; (2) Boris Klain, Borok Geophysical Observatory of Schmidt Institute of Physics of the Earth, Russian Academy of Sciences; (3) Alexey Zavyalov, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences; (4) Oleg Zotov, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences and Borok Geophysical Observatory of Schmidt Institute of Physics of the Earth, Russian Academy of Sciences. Authors: Authors: (1) Anatol Guglielmi, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences; (2) Boris Klain, Borok Geophysical Observatory of Schmidt Institute of Physics of the Earth, Russian Academy of Sciences; (3) Alexey Zavyalov, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences; (4) Oleg Zotov, Schmidt Institute of Physics of the Earth, Russian Academy of Sciences and Borok Geophysical Observatory of Schmidt Institute of Physics of the Earth, Russian Academy of Sciences. Table of Links Abstract and Introduction Abstract and Introduction Omori (1894) Omori (1894) Hirano (1924) Hirano (1924) Discussion Discussion Conclusion and References Conclusion and References Omori (1894) It is best to start describing the outstanding events of a century ago from afar. In 1850, John Milne was born in Liverpool, who in the future was to become a famous geophysicist, one of the founders of modern seismology [15]. He was educated in London, worked as a mining engineer in Newfoundland, as a geologist in the Sinai Peninsula, and from 1875 to 1895 worked in Tokyo at the invitation of the government of the Japanese Empire. It is curious that in search of adventure, Milne traveled to Tokyo for three months, mostly by land (via Siberia). In 1880, he created a horizontal pendulum seismograph - the first easy-to-use and sufficiently sensitive instrument for recording earthquakes. In 1887 Milne was elected a member of the Royal Society. He manages to convince the Society to allocate funds for the creation of a world network of seismic stations equipped with his instruments. Japan highly appreciated Milne's services to the country and the world. Emperor Meiji awarded him the Order of the Rising Sun and awarded him a lifetime pension of 1,000 yen. The University of Tokyo elected him honorary professor. Fusakichi Omori was a student of John Milne. He enjoyed the encouraging support of his teacher, as did all young Japanese seismologists of that time. On October 28, 1891, the earthquake with a magnitude of M = 8 occurred. Milne's seismographs recorded numerous aftershocks. Analysis of these aftershocks allowed Omori to formulate the law that bears his name [2]. It is worth mentioning that he was 26 years old at the time. Omori's Law states that after a strong earthquake, the frequency of aftershocks, i.e. tremors following the main shock, on average, decreases hyperbolically over time: Here k > 0, c > 0, t ≥ 0. This was the first law of earthquake physics, if we keep in mind the chronological sequence of outstanding discoveries in seismology. Intuition tells us that the Omori formula (6) has the characteristics of a fundamental law of nature. Let's explain what was said. Firstly, (6) can be rewritten in the equivalent form n kt = /, where k > 0, t > 0. In fact, contrary to popular belief, the parameter in (6) is free due to the homogeneity of the flow of time t. In other words, Omori’s law is one-parameter: the value of k generally characterizes the source of an earthquake over the entire history of the evolution of aftershocks. The general solution of equation (2) is Taking (1) into account, it is easy to see that Omori’s law (6) follows from (7) under the condition s = const . In this case k =1/σ , 0 c n =1/ σ . In other words, Omori's law is satisfied provided that the deactivation coefficient remains unchanged throughout the entire history of aftershock excitation. The found condition for the applicability of the Omori law is tough. Is it carried out experimentally? To answer the question, we compiled the Aftershock Atlas [8], which presents solutions to the inverse source problem. The characteristic size of the source was determined according to the method [16]. Analysis of solutions for several dozen events indicates that in all cases, at the first stage of the evolution of aftershocks, immediately after the main shock, the source deactivation coefficient remains unchanged within the limits of measurement accuracy. We call the time interval during which s = const the Epoch of Hyperbolic Evolution (EHE) , or Omori epoch. The duration of EHE varies from case to case from a few days to several months. Figure 1 shows a case of a fairly long EHE, almost equal to 100 days. There is a tendency for σ to decrease with increasing magnitude of the main shock [17]. The discovery of the Omori epoch is extremely significant. At the end of the epoch, the state of the source changes. Let's introduce parameter θ = dσ / dt. In the Omori epoch, θ = 0. The transition to a new state is indicated by a sharp jump in parameter θ (Figure 2). This suggests that the end of the Omori epoch is accompanied by a bifurcation of the earthquake source. θ = dσ / dt. θ θ A convincing illustration of this is Figure 3. It shows the result of processing and analyzing eight events. The transition from one mode of source deactivation to a qualitatively different mode is abrupt in the sense that its duration is much less than the duration of the Omori epoch. Let us summarize the preliminary results of studying the evolution of aftershocks by solving the inverse problem: 1The Omori formula (6) has the status of a law that is fulfilled with good accuracy in a real geophysical process. 1The Omori formula (6) has the status of a law that is fulfilled with good accuracy in a real geophysical process. The applicability of Omori's law is strictly limited to the epoch of harmonic evolution of the earthquake source. At the end of epoch of harmonic evolution, the source experiences a bifurcation. The applicability of Omori's law is strictly limited to the epoch of harmonic evolution of the earthquake source. The applicability of Omori's law is strictly limited to the epoch of harmonic evolution of the earthquake source. At the end of epoch of harmonic evolution, the source experiences a bifurcation. At the end of epoch of harmonic evolution, the source experiences a bifurcation. To conclude the section devoted to Fusakichi Omori, let us return to formulas (6) and (7). The law of aftershocks (7) formally differs from law (6) only in that in it time t is replaced by proper time τ . The concept of proper time and the corresponding term were introduced into physics only at the beginning of the 20th century. It would seem that Omori simply did not have the opportunity to use the concept of proper time in order to give formula (6) the flexibility and physical meaning that our formula (7) has. In this regard, we want to clarify that the source proper time (1) is not directly related to the relativistic proper time. Formulas of the form (1) express the uneven functioning of any dynamic system immersed in a nonstationary environment or located in nonstationary external force fields. In our case, formula (1) takes into account the non-stationary state of the source that experienced the main shock of the earthquake. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Two Memorable Dates in Seismology: Omori (1894)

Introduction In this article, I'll address a critical aspect of TCP communication: effectively managing scenarios where the server fails to respond. I focus on a specific scenario where the application only sends data over TCP without receiving any application-level response from the server. This exploration covers TCP communication from the application's perspective, highlighting both the application layer and the underlying OS operations. You'll learn how to set effective timeouts to avoid data loss during unresponsive server instances. I’ll provide code examples in Ruby, but the idea stays the same for any language. The Challenge of Silent TCP Servers Imagine you're working with an application that consistently transmits data over a TCP socket. While TCP is designed to ensure packet delivery at the transport level within the defined TCP stack configurations, it's interesting to consider what this implies at the application level. To better understand this, let's construct a sample TCP server and client using Ruby. This will allow us to observe the communication process in action. : server.rb # server.rb
require 'socket'
require 'time'

$stdout.sync = true

puts 'starting tcp server...'

server = TCPServer.new(1234)

puts 'started tcp server on port 1234'

loop do
  Thread.start(server.accept) do |client|
    puts 'new client'
    while (message = client.gets)
      puts "#{Time.now}]: #{message.chomp}"
    end
    client.close
  end
end And the : client.rb require 'socket'
require 'time'

$stdout.sync = true

socket = Socket.tcp('server', 1234)

loop do
  puts "sending message to the socket at #{Time.now}"
  socket.write "Hello from client\n"

  sleep 1
end And let’s encapsulate this setup in containers using this : Dockerfile FROM ruby:2.7

RUN apt-get update && apt-get install -y tcpdump

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy the current directory contents into the container at /usr/src/app
COPY . . and : docker-compose.yml version: '3'

services:
  server:
    build:
      context: .
      dockerfile: Dockerfile
    command: ruby server.rb
    volumes:
      - .:/usr/src/app
    ports:
      - "1234:1234"
    healthcheck:
      test: ["CMD", "sh", "-c", "nc -z localhost 1234"]
      interval: 1s
      timeout: 1s
      retries: 2
    networks:
      - net

  client:
    build:
      context: .
      dockerfile: Dockerfile
    command: ruby client.rb
    volumes:
      - .:/usr/src/app
      - ./data:/data
    depends_on:
      - server
    networks:
      - net

networks:
  net: Now, we can easily run this with and see in logs how the client sends the message and the server receives it: docker compose up $ docker compose up
[+] Running 2/0
 ⠿ Container tcp_tests-server-1  Created                                                                                                                                                                         0.0s
 ⠿ Container tcp_tests-client-1  Created                                                                                                                                                                         0.0s
Attaching to tcp_tests-client-1, tcp_tests-server-1
tcp_tests-server-1  | starting tcp server...
tcp_tests-server-1  | started tcp server on port 1234
tcp_tests-client-1  | sending message to the socket at 2024-01-14 08:59:08 +0000
tcp_tests-server-1  | new client
tcp_tests-server-1  | 2024-01-14 08:59:08 +0000]: Hello from client
tcp_tests-server-1  | 2024-01-14 08:59:09 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-14 08:59:09 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 08:59:10 +0000
tcp_tests-server-1  | 2024-01-14 08:59:10 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-14 08:59:11 +0000
tcp_tests-server-1  | 2024-01-14 08:59:11 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-14 08:59:12 +0000
tcp_tests-server-1  | 2024-01-14 08:59:12 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-14 08:59:13 +0000 Pretty easy so far, huh? However, the situation gets more interesting when we simulate a server failure for an active connection. We do this using : docker compose stop server tcp_tests-server-1  | 2024-01-14 09:04:23 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:24 +0000
tcp_tests-server-1  | 2024-01-14 09:04:24 +0000]: Hello from client
tcp_tests-server-1 exited with code 1
tcp_tests-server-1 exited with code 0
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:25 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:26 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:27 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:28 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:29 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:30 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:31 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:04:32 +0000 We observe that the server is now offline, yet the client behaves as if the connection is still active, continuing to send data to the socket without hesitation. This leads me to question why this occurs. Logically, the client should detect the server's downtime within a short span, possibly seconds, as TCP fails to receive acknowledgments for its packets, prompting a connection closure. However, the actual outcome diverged from this expectation: tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:11 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:12 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:13 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:14 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:15 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:16 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:17 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:18 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:19 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:20 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:21 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:22 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-14 09:20:23 +0000
tcp_tests-client-1  | client.rb:11:in `write': No route to host (Errno::EHOSTUNREACH)
tcp_tests-client-1  | 	from client.rb:11:in `block in <main>'
tcp_tests-client-1  | 	from client.rb:9:in `loop'
tcp_tests-client-1  | 	from client.rb:9:in `<main>'
tcp_tests-client-1 exited with code 1 In reality, the client may remain unaware of the disrupted connection for as long as ! 15 minutes What causes this delay in detection? Let's delve deeper to understand the reasons. In-Depth: TCP Communication Mechanics To fully cover this case, let's first revisit the basic principles, followed by an examination of how the client transmits data over TCP. TCP basics Here is the basic diagram illustrating the TCP flow: Once a connection is established, the transmission of each message typically involves two key steps: The client sends the message, marked with the PSH (Push) flag. The server acknowledges receipt by sending back an ACK (Acknowledgment) response. Communication Between the Application and Socket Below is a simplified sequence diagram illustrating the opening of a TCP socket by an application and the subsequent data transmission through it: The application makes two operations: Opening a TCP socket Sending data to the open socket For instance, when opening a TCP socket, as done with Ruby's command, the system synchronously creates a socket using the system call and then establishes a connection via the system call. Socket.tcp(host, port) socket(2) connect(2) As for sending data, using a command like in an application primarily places the message into the socket's send buffer. It then returns the count of bytes that were successfully queued. The actual transmission of this data over the network to the destination host is managed asynchronously by the TCP/IP stack. socket.write('foo') This means that when an application writes to the socket, it isn't directly involved in the network operations and might not know in real-time if the connection is still active. The only confirmation it receives is that the message has been successfully added to the TCP send buffer. What Happens When the TCP Server Goes Down? As the server does not respond with an ACK flag, our TCP stack will initiate a retransmission of the last unacknowledged packet: The interesting thing here is that by default TCP makes 15 retransmissions with exponential backoff which results in almost 15 minutes of retries! You can check how many retries are set on your host: $ sysctl net.ipv4.tcp_retries2
net.ipv4.tcp_retries2 = 15 After diving into docs, it’s becoming clear; documentation says: ip-sysctl.txt The default value of 15 yields a hypothetical timeout of 924.6 seconds and is a lower bound for the effective timeout. TCP will effectively time out at the first RTO which exceeds the hypothetical timeout. During this period, the local TCP socket is alive and accepts data. When all retries are made, the socket is closed, and the application receives an error on an attempt to send anything to the socket. Why Is It Usually Not an Issue? The scenario where the TCP server unexpectedly goes down without sending FIN or RST TCP flags, or when there are connectivity issues, is quite common. So why do such situations often go unnoticed? Because, in most cases, the server responds with some response on the application level. For example, the HTTP protocol requires the server to respond to every request. Basically, when you have code like , it makes two main operations: connection.get Writes your payload to the TCP socket's send buffer. From this point onward, the operating system's TCP stack takes responsibility for reliably delivering these packets to the remote server with TCP guarantees. Waiting for a response in a TCP receiving buffer from the remote server Typically, applications use nonblocking reads from the receiving buffer of the same TCP socket. This approach simplifies matters considerably because, in such cases, we can easily set a timeout on the application level and close the socket if there is no response from the server within a defined time frame. However, when we don't expect any response from the server (except TCP acknowledgments), it becomes less straightforward to determine the connection's status from the application level The Impact of Long TCP Retransmissions So far, we've established the following: The application opens a TCP socket and regularly writes data to it. At some point, the TCP server goes down without even sending an RST packet, and the TCP stack of the sender begins retransmitting the last unacknowledged packet. All other packets written to that socket are queued in the socket’s send buffer. By default, the TCP stack attempts to retransmit the unacknowledged packet 15 times, employing exponential backoff, resulting in a duration of approximately 924.6 seconds (about 15 minutes). During this 15-minute period, the local TCP socket remains open, and the application continues to write data to it until the send buffer is full (which typically has a limited capacity, often just a few megabytes). When the socket is eventually marked as closed after all retransmissions, . all the data in the send buffer is lost This is because the application is no longer responsible for it after writing to the send buffer, and the operating system simply discards this data. The application can only detect that the connection is broken when the send buffer of the TCP socket becomes full. In such cases, attempting to write to the socket will block the main thread of the application, allowing it to handle the situation. However, the effectiveness of this detection method depends on the size of the data being sent. For instance, if the application sends only a few bytes, such as metrics, it may not fully fill the send buffer within this 15-minute timeframe. So, how can one implement a mechanism to close a connection with an explicitly set timeout when the TCP server is down to avoid 15 minutes of retransmissions and losing of data during this period? TCP Retransmissions Timeout Using the Socket Option In private networks, extensive retransmissions aren't typically necessary, and it's possible to configure the TCP stack to attempt only a limited number of retries. However, this configuration applies globally to the entire node. Since multiple applications often run on the same node, altering this default value can lead to unexpected side effects. A more precise approach is to set a retransmission timeout exclusively for our socket using the socket option. By employing this option, the TCP stack will automatically close the socket if retransmissions do not succeed within the specified timeout, regardless of the maximum number of TCP retransmissions set globally. TCP_USER_TIMEOUT On the application level, this results in an error being received upon attempting to write data to a closed socket, allowing for proper data loss prevention handling. Let’s set this socket option in : client.rb require 'socket'
require 'time'

$stdout.sync = true

socket = Socket.tcp('server', 1234)

# set 5 seconds restransmissions timeout
socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_USER_TIMEOUT, 5000)

loop do
  puts "sending message to the socket at #{Time.now}"
  socket.write "Hello from client\n"

  sleep 1
end Also, from my observations, the TCP_USER_TIMEOUT socket option is not available on macOS. Now, start everything again with , and at some point, let’s stop the server again with : docket compose up docker compose stop server $ docker compose up
[+] Running 2/0
 ⠿ Container tcp_tests-server-1  Created                                                                                                                                                                         0.0s
 ⠿ Container tcp_tests-client-1  Created                                                                                                                                                                         0.0s
Attaching to tcp_tests-client-1, tcp_tests-server-1
tcp_tests-server-1  | starting tcp server...
tcp_tests-server-1  | started tcp server on port 1234
tcp_tests-server-1  | new client
tcp_tests-server-1  | 2024-01-20 12:37:38 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:38 +0000
tcp_tests-server-1  | 2024-01-20 12:37:39 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:39 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:40 +0000
tcp_tests-server-1  | 2024-01-20 12:37:40 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:41 +0000
tcp_tests-server-1  | 2024-01-20 12:37:41 +0000]: Hello from client
tcp_tests-server-1  | 2024-01-20 12:37:42 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:42 +0000
tcp_tests-server-1  | 2024-01-20 12:37:43 +0000]: Hello from client
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:43 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:44 +0000
tcp_tests-server-1  | 2024-01-20 12:37:44 +0000]: Hello from client
tcp_tests-server-1 exited with code 1
tcp_tests-server-1 exited with code 0
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:45 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:46 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:47 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:48 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:49 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:50 +0000
tcp_tests-client-1  | sending message to the socket at 2024-01-20 12:37:51 +0000
tcp_tests-client-1  | client.rb:11:in `write': Connection timed out (Errno::ETIMEDOUT)
tcp_tests-client-1  | 	from client.rb:11:in `block in <main>'
tcp_tests-client-1  | 	from client.rb:9:in `loop'
tcp_tests-client-1  | 	from client.rb:9:in `<main>'
tcp_tests-client-1 exited with code 1 At ~12:37:45, I stopped the server, and we saw that the client got almost in 5 seconds, great. Errno::ETIMEDOUT Let’s capture a tcpdump with : docker exec -it tcp_tests-client-1 tcpdump -i any tcp port 1234 It's worth noting that the timeout actually occurs in a bit more than 5 seconds. This is because the check for exceeding TCP_USER_TIMEOUT happens at the next retry. When the TCP/IP stack detects that the timeout has been exceeded, it marks the socket as closed, and our application receives the error Errno::ETIMEDOUT Also, if you are using TCP keepalives, I recommend checking out this . It covers the nuances of using TCP_USER_TIMEOUT in conjunction with TCP keepalives. article from Cloudflare

Control TCP Retransmissions: Early Issue Detection to Prevent Data Loss

টিসিপি রিট্রান্সমিশন নিয়ন্ত্রণ করুন: ডেটা ক্ষতি রোধ করতে প্রাথমিক সমস্যা সনাক্তকরণ

Kontrollieren Sie TCP-Neuübertragungen: Frühzeitige Problemerkennung, um Datenverlust zu verhindern

Controle las retransmisiones TCP: detección temprana de problemas para evitar la pérdida de datos

Contrôler les retransmissions TCP : détection précoce des problèmes pour éviter la perte de données

टीसीपी रीट्रांसमिशन को नियंत्रित करें: डेटा हानि को रोकने के लिए प्रारंभिक समस्या का पता लगाना

TCP 再送信の制御: 問題を早期に検出してデータ損失を防止

TCP 재전송 제어: 데이터 손실을 방지하기 위한 조기 문제 감지

Controle retransmissões TCP: detecção precoce de problemas para evitar perda de dados

Управление повторными передачами TCP: раннее обнаружение проблем для предотвращения потери данных

TCP Yeniden İletimlerini Kontrol Edin: Veri Kaybını Önlemek için Sorunun Erken Tespiti

Kiểm soát việc truyền lại TCP: Phát hiện vấn đề sớm để ngăn ngừa mất dữ liệu

控制 TCP 重传：早期问题检测以防止数据丢失

My research journey into ML in Elixir has been marked by the discovery of Elixir's unique strengths and its potential to revolutionize how ML workflows are designed and executed. The fusion of Elixir's concurrency, fault tolerance, and scalability with the power of ML opens up exciting possibilities for building robust and efficient ML systems. As I continue to explore this fusion, I am eager to witness how Elixir's functional elegance can shape the future of machine learning.  In this article, we look at a few examples of using the Elixir language in the context of machine learning. Machine Learning With Elixir, Nx, and Axon In this example, we'll create a simple Elixir application to perform linear regression using the Nx library for numerical operations and Axon for machine learning. First, ensure you have the necessary dependencies in your file: mix.exs defp deps do
  [
    {:axon, "~> 0.8"},
    {:nx,   "~> 2.0"} 
  ]
end Now, let's create a module for linear regression: defmodule LinearRegression do
  use Axon.Model

  defstruct [:slope, :intercept]

  def init(_opts) do
    %{slope: 0.0, intercept: 0.0}
  end

  def fit(data) do
    x_values = Nx.get_column(data, 0)
    y_values = Nx.get_column(data, 1)

    x_mean = Nx.mean(x_values)
    y_mean = Nx.mean(y_values)

    numerator = Nx.dot(x_values - x_mean, y_values - y_mean)
    denominator = Nx.dot(x_values - x_mean, x_values - x_mean)

    slope = numerator / denominator
    intercept = y_mean - slope * x_mean

    %{slope: slope, intercept: intercept}
  end

  def predict(model, x) do
    model.slope * x + model.intercept
  end
end Let's use this module to perform linear regression: data = Nx.Matrix.new([[1.0, 2.0], [2.0, 3.0], [3.0, 4.0]])
model = LinearRegression.fit(data)
prediction = LinearRegression.predict(model, 4.0)

IO.puts("Predicted value for x = 4.0: #{prediction}") In this example, we created a simple Elixir application to perform linear regression using the Nx for numerical operations and Axon for machine learning. Let's add complexity and create a small example of machine learning. Machine Learning and ETL With Elixir Elixir is known for its ability to create highly concurrent and distributed systems. We can use these features to create machine learning and ETL processes. For this example, let's assume we have a large dataset that we want to process in parallel using multiple nodes. We'll also use Elixir's immutable data structures for data transformation. Before we start, we need to add the necessary dependencies to our file: mix.exs defp deps do
  [
    {:flow, "~> 1.0"},
    {:axon, "~> 0.8"},
    {:enum_actor, "~> 0.5"}
  ]
end Start multiple Elixir nodes, each with a unique name. You can do this by running the following command in separate terminals: # Node 1
iex --sname node1 -S mix

# Node 2
iex --sname node2 -S mix

# Node 3
iex --sname node3 -S mix Define a distributed ETL process: defmodule DistributedETL do
  def generate_data(node) do
    # Simulate data extraction (e.g., from a distributed source)
    {:ok, data} = Flow.from_enum(1..10)
    data
    |> Flow.partition_by(:local, 2, fn _ -> node end)
    |> Flow.map(&ETL.process_data(&1))
    |> Flow.flat_map(&Enum.shuffle/1)
    |> Flow.into([])
    |> Flow.run()
  end
end Define ETL operations: defmodule ETL do
  def process_data(data) do
    # Simulate data transformation (e.g., filtering, mapping, feature engineering)
    Enum.map(data, &(&1 * 2))  # Double each value
  end
end Distribute the ETL process, on each node, run the distributed ETL process, and collect the results: # On Node 1
result_node1 = DistributedETL.generate_data(:node1)

# On Node 2
result_node2 = DistributedETL.generate_data(:node2)

# On Node 3
result_node3 = DistributedETL.generate_data(:node3) Aggregate and perform ML; we can now aggregate the results from all nodes and perform machine learning on the combined data: all_results = result_node1 ++ result_node2 ++ result_node3

# Train a machine learning model (as shown in the previous example)
trained_model = LinearRegression.fit(all_results)

# Store or use the trained model as needed
IO.inspect(trained_model) Ensure that all nodes have the necessary code loaded, and run the pipeline on each node separately. This example demonstrates a simplified distributed ETL and machine learning pipeline across multiple nodes in Elixir. In a real-world scenario, you would need to consider more complex data distribution, fault tolerance, and load balancing, possibly using a distributed task coordination system like GenStage or Flow. But an even more interesting trick can be done by taking advantage of the fact that you can train and update the model in a running application. Real-time Training Here's a simplified example of real-time model training using a GenStage producer and Axon for linear regression. # Define a GenStage producer for data ingestion
defmodule RealTimeDataProducer do
  use GenStage

  def init(_) do
    {:producer, nil}
  end

  def handle_demand(demand, state) do
    new_state = fetch_real_time_data()
    {:noreply, new_state, new_state}
  end

  defp fetch_real_time_data() do
    # Simulate fetching real-time data
    new_data = ...  # Fetch data from a source
    {:producer, new_data}
  end
end # Model training process
defmodule RealTimeModelTraining do
  use GenStage

  def init(state) do
    {:consumer, state}
  end

  def handle_events(events, state) do
    # Train the machine learning model using Axon
    updated_model = train_model(state, events)
    {:noreply, updated_model, updated_model}
  end

  defp train_model(model, events) do
    # Perform incremental model training with new data
    Enum.reduce(events, model, fn event, acc ->
      # Update the model based on new data
      updated_model = LinearRegression.online_fit(acc, event)
      updated_model
    end)
  end
end Deploy the trained model for making predictions on new data as it arrives. This could be done within the same Elixir application or through a separate service. Keep in mind that building a production-ready real-time machine learning system involves various considerations, including data quality, model versioning, monitoring, and scalability. You may also need to use distributed stream processing frameworks for handling high-volume real-time data streams effectively. Elixir's machine-learning libraries are being actively developed and will provide even greater capabilities and applications in a variety of fields. I believe that Elixir has a long and exciting journey ahead of it in the world of machine learning.

Machine Learning: ETL and Real-Time Training With Elixir

1. Introduction The project is a simple example showcasing a quick way to set up an Arduino with a USB Host Shield as a USB CDC Host capable of communicating with the BleuIO Dongle. When a BleuIO Dongle is connected to the USB port, the BleuIO Dongle will start advertising. It will then act as a terminal, taking input and sending data to the Arduino Virtual Com Port. We have used an Arduino Uno Rev. 3 with SparkFun's USB Host Shield (DEV-09947) for this example. 2. About the Code You can get project HERE This project is based on the 'acm_terminal' example in the Host USB Shield Library 2.0 The largest possible max.packet size for the function Acm.RcvData() is 64 bytes, so to accommodate the amount of data we will receive, we are using three buffers to receive the data from the BleuIO Dongle. If the buffers have received any data, we print it out to the serial terminal connected to the Virtual COM Port. void loop()
{
    Usb.Task();

    if( Acm.isReady()) {
       uint8_t rcode;
       uint8_t rcode2;
       uint8_t rcode3;

       /* reading the keyboard */
       if(Serial.available()) {
         uint8_t data= Serial.read();
         /* sending to the BleuIO Dongle */
         rcode = Acm.SndData(1, &data);
         if (rcode)
            ErrorMessage<uint8_t>(PSTR("SndData"), rcode);
       }//if(Serial.available()...

        if(start_flag == 0x00)
        {
            rcode = Acm.SndData(strlen((char *)START_CMDS), (uint8_t *)START_CMDS);
            if (rcode)
            {
                ErrorMessage<uint8_t>(PSTR("SndData"), rcode);
            }

          start_flag = 0x01;
        }
        /* reading the BleuIO Dongle */
        uint8_t  buf[64];
        uint16_t rcvd = 64;
        uint8_t  buf2[64];
        uint16_t rcvd2 = 64;
        uint8_t  buf3[64];
        uint16_t rcvd3 = 64;
        uint8_t  dongle_input[3*64];
        uint16_t input_indx = 0;

        memset(dongle_input, 0, sizeof(dongle_input));

        rcode = Acm.RcvData(&rcvd, buf);
        delay(1);
        rcode2 = Acm.RcvData(&rcvd2, buf2);
        delay(1);
        rcode3 = Acm.RcvData(&rcvd3, buf3);
         if (rcode && rcode != hrNAK)
         {
            ErrorMessage<uint8_t>(PSTR("Ret"), rcode);
         }

         if (rcode2 && rcode2 != hrNAK)
         {
             ErrorMessage<uint8_t>(PSTR("Ret"), rcode2);
         }

         if (rcode3 && rcode3 != hrNAK)
         {
             ErrorMessage<uint8_t>(PSTR("Ret"), rcode3);
         }


            if( rcvd ) { //more than zero bytes received
              for(uint16_t i=0; i < rcvd; i++ ) {
                Serial.print((char)buf[i]); //printing on the screen
                dongle_input[input_indx] = buf[i];
                input_indx++;
              }
            }

            if( rcvd2 ) { //more than zero bytes received
              for(uint16_t i=0; i < rcvd2; i++ ) {
                Serial.print((char)buf2[i]); //printing on the screen
                dongle_input[input_indx] = buf2[i];
                input_indx++;
              }
            }

            if( rcvd3 ) { //more than zero bytes received
              for(uint16_t i=0; i < rcvd3; i++ ) {
                Serial.print((char)buf3[i]); //printing on the screen
                dongle_input[input_indx] = buf3[i];
                input_indx++;
              }
            }
            dongle_input[input_indx] = 0x00;

            // Example on a way for the Arduino to react to BleuIO events
            if(strlen((char *)dongle_input) != 0)
            {
              if(strstr((char *)dongle_input, "handle_evt_gap_connected") != NULL)
              {
                Serial.print("<<CONNECTION DETECTED!>>");
              }
              else if(strstr((char *)dongle_input, "handle_evt_gap_disconnected") != NULL)
              {
                Serial.print("<<CONNECTION LOST!>>");
              }
            }
    }//if( Usb.getUsbTaskState() == USB_STATE_RUNNING..
} We also store the latest data from the dongle into the dongle_input buffer and run it through a simple "parser" to showcase an easy way of how you can react to events and have the Arduino do something. In this example, we are explicitly looking for BLE connection or disconnect events. When found, we just print out "<<CONNECTION DETECTED!>>" or "<<CONNECTION LOST!>>" to the terminal. 3. Using the example project 3.1 What you will need A BleuIO dongle with firmware version 2.1.0 or later ( ) https://www.bleuio.com/getting_started/docs/release_history/#release-v210 An Arduino Uno Rev. 3 ( ) https://store.arduino.cc/products/arduino-uno-rev3 An USB Host Shield ( ) https://www.sparkfun.com/products/9947 The Arduino IDE ( ) https://www.arduino.cc/en/software 3.2 Requirments for the SparkFun board With the SparkFun board, it seems like you MUST supply external power on Vin or the barrel jack. 5V from the USB cable did not work. You must also apply a jumper from pin D7 to RESET. 4. How to setup project 4.1 Downloading the project from GitHub Get project HERE https://github.com/smart-sensor-devices-ab/arduino_bleuio_example Either clone the project, or download it as a zip file and unzip it, into your Arduino folder. 4.2 Downloading the USB Host Shield Library 2.0 Either download the Library from Here ( ) and place the folder into your libraries folder inside your Arduino folder. https://felis.github.io/USB_Host_Shield_2.0/ (For information on installing libraries, see: ) http://www.arduino.cc/en/Guide/Libraries Or download it through the Arduino IDE: In Arduino IDE choose Sketch>Include Library>Manage Library Search for USB Host Shield Library 2.0 and click 'Install' 5. Running the example In Arduino IDE click the upload button to upload the project to your Arduino. Open up the 'Arduino Uno Viritual COM Port' with a serial terminal emulation program like TeraTerm, Putty or CoolTerm. Serial port Setup: Baudrate: 115200 Data Bits: 8 Parity: None Stop Bits: 1 Flow Control: None Or inside the Arduino IDE open up Arduino Monitor and in the bottom right corner select 'Carriage Return' and '115200 baud' You should see the word 'Start' and then see the dongle running two commands: setting response data and starting the advertising. You can now type commands to the dongle.

Using Arduino for Bluetooth Low Energy (BLE): A Step-by-Step Guide

Cheapskate's Journey to On-Demand Load Tests with Locust

Anyone who has traded or invested in Bitcoin stocks before has been frustrated by the difficulty involved with trying to predict market trends. cryptocurrencies Usually, this leads people to follow the consensus of the public, or their favored crypto experts, or even the advice of online trading communities on Telegram, Discord, or Reddit. None of these are bad options, so long as a trader finds trustworthy people. The best way to , however, might not involve many ‘people’ at all. figure out trends What I’m referring to is to is the growing prevalence of a technique known as machine learning. Through , blockchain based currencies become slightly more predictable in their trends, if no less volatile, and are able to be safer, more stable systems than they were prior to its various implementations. machine learning What is Machine Learning? Machine learning, a term originally attributed to the late Arthur Samuel around 1959, is an exciting technology which is the basis of much of the personalization users are able to experience on the web today. While machine learning in some form or another has existed for quite a long time, modern day is a whole different beast. machine learning It operates from the theory that machines are able to learn to do certain things, recognizing and analyzing certain patterns and behaviors, without having to be painstakingly programmed for each microscopically specific part of the task. In essence, it's the idea of computers learning from data sets. As machines are exposed to more and more data, they would be able to adapt to it without, or at least with minimal outside interference, growing smarter and more capable in their roles. This sort of constant self improvement requires the ability to run numerous complex calculations, automatically, repeatedly, over relatively short periods of time. Not having access to this level of processing power held back the machine learning of the past, but in modern times this no longer presents as much of a problem. Nowadays, machine learning algorithms are almost ever present in our digital lives. Video recommendations, tailored ads, even fraud detection. All of this is made possible through creative implementation of machine learning, coupled with incredibly large quantities of data sets. Of course a tool this powerful was eventually going to find its way to being used in financial markets. The information it could give would be invaluable. Below we take a look at four of the ways machine learning is implemented in the cryptocurrency markets. The Good 1. Flow Analysis The single greatest, most effective way of determining the future performance of a given asset is through the analysis of the flow of its funds. By examining the way funds are being transferred by known entities and comparing that to previously known data sets, machine learning is able to help predict value shifts in an asset. What does this mean, in a practical sense? Well, it means traders are able to determine what big players in the cryptocurrency markets are doing. For instance, whales might be working towards the completion of a the value of an asset. pump and dump scheme, which would massively decrease Knowing this, an investor would be able to jump ship before catastrophe struck. Maybe instead of preparing a dump, whales have begun accumulating an asset that recently had a downtrend. This could allow an investor to get in on the ground floor before the asset significantly increased in value. The applications of this are numerous in the crypto space, making this type of machine learning something that will be absolutely vital to successful trading down the line. 2. Address Classification While it might seem minor, the use of machine learning in classifying is a powerful tool. wallet addresses By identifying which wallet addresses are exchange wallets and individual wallets, machine learning models can learn the behavior of crypto exchanges, where previously this would be impossible due to the lack of comprehensive data sets. 3. Analyzing Trading Behaviors A first in a new and perhaps innovative way of trying to understand how assets perform in the cryptocurrency markets, recurrent neural networks are being used to better understand and predict the trading patterns of specific investors. In a given set of investors, this type of machine learning is used to identify investors in groups and discover the way they invest their capital, the patterns they follow. This data can then be used to actually predict how they will invest in the future. Its very difficult to apply this kind of learning to an asset itself, so an understanding of the asset can instead be formed through analysis of how individuals interact with it. 4. Fraud Detection Among the most useful ways machine learning is being put to use in the cryptosphere is as a way to detect . fraudulent transactions The nature of cryptocurrencies allows the traders that use them to be more or less anonymous. Transactions are nearly instantaneous, and the information that gets recording into the public ledger isn’t really identifiable data. In addition, any damage the fraudsters manage to create is permanent, with the only way to reverse it being to create a fork in the blockchain, effectively creating an entirely different cryptocurrency. These all create a set of circumstances which requires a far approach to stopping fraud than would be applied to fiat currencies. Because of these unique challenges, major exchanges such as Coinbase have had to implement unique solutions. This is where machine learning comes in. They tackle this problem by using a prevention system consisting of both machines and human analysts, supporting each other in feedback loop. The machines start by being trained on well-known fraudulent patterns under a variety of conditions, allowing them to generalize the knowledge and identify when similar patterns are found. The humans ensure everything stays accurate, and in doing so make the machines better at identifying the correct patterns. This has a huge affect on individual trader. By creating a safer, more secure marketplace for investors to trade their cryptocurrencies within, exchanges like Coinbase ensure that cryptocurrencies remain a viable concept. They also are able to somewhat stabilize the markets, since the machine learning algorithm is able to discourage the practices of groups from becoming so ever prevelant that they might wreck the blockchain currency ecosystem as a whole. pump and dump The Bad As we’ve seen, machine learning in the cryptosphere is going strong. Its being used as a powerful tool to determine how individual and exchanges will invest, and what trends the markets is facing right now. Machine learning is even being used as a powerful anti fraud tool. Ironic, then, that the way most people will hear about machine learning in the cryptocurrency field is itself a vast misrepresentation of facts. The devil I speak of is those many ‘price prediction’ models a trader will stumble upon while researching the legitimate ways machine learning is being utilized. At first many of these will seem quite convincing. They’ll show you a picture of a graph of some sort, comparing the outcome of the machines predictions to the actual data from the market. These are shown to match up incredibly well. Unbelievably well, you might even venture to say. You would be more correct than you know. While the work the teams behind these machine put into their creations is certainly admirable, insinuating in any way that these should be used as a method of prediction in trading on the real market is intellectually dishonest, to say the least. learning algorithms If you take a close look at these graphs, you usually find that the prediction lines matches up to the factual line very closely. In fact, they are nearly identical, except that the predictions are pushed out by one day. The machine is essentially learning how the asset performed the previous day and translating that over, due to flaws in the methodology the models were based around. The models don’t actually present real correlation between their predictions and the true results. In the future there may be models that can accurately predict how an asset will perform day to day, but in our current time this just isn’t possible. To avoid potentially massive hits to your portfolio you should avoid trading off of the advice of these particular kinds of machine learning models. Final Thoughts While you shouldn’t trust every article you read on the internet, there are still a fair number of interesting and powerful applications of machine learning technology currently being used in cryptocurrency today. As both blockchain and machine learning continue to grow, we will inevitably continue to see incredible innovation in both fields, which will certainly allow investors to better understand the markets they trade in.

Applying Machine Learning to Crypto-Sphere: The Good and the Bad Aspects

Share Code between React and React Native Apps

Chart structure Splitting grid and traces Grid Color Detection Trace Pixel Classification Chart template detection Text Extraction Text recognition Template matching Axis Matching Pen Trajectory Evaluation Shear compensation Improving envelope Outlier removal Pen trajectory evaluation Conclusion Flow charts are widely used in gas industry. They record parameters of a gas flow (like pressure) on a paper disc. This data is then used to calculate gas volume passed through a pipe. The chart recorder contains two or three pens driven by sensors. These pens plot traces on a rotating paper disk. The disk performs one revolution per day or week, rarely per month. The disk is replaced after one revolution. Currently disks are processed manually. The operator uses the vector graphics editor to convert the scanned bitmap to contours and then to tabular data. This approach relies heavily on the operator’s attention and experience. There is a high probability of mistakes. After all, such job is really boring! We have developed an algorithm that performs this job automatically. Given a bitmap with a scanned chart, it produces tabular representation of chart data. Chart structure The template on the disc (usually green) is pre-drawn. It contains: Radial arcs for time; Concentric circles for quantity levels; Names of quantities (axes); Quantity values for concentric lines (ranges); Time marks: hours and week day names. Note that the same concentric line has different values for different axes. Splitting grid and traces Grid Color Detection The first task to solve is to separate trace pixels and pixels belonging to the blank area. This blank area contains two dominant colors: white and green. In theory, knowing this we could use simple thresholding to detect these two colors. But in practice this approach drops pixels on the edge between the grid and the blank paper. This happens because boundary pixels aren’t strictly white or green — they are somewhere between. To deal with it, a linear model is used. To build such model, we need an area where no trace appears. Such area is the outer ring of the chart, named “Time area” on the image. We take histograms for red-green, green-blue and blue-red correspondence and then build linear models for each. After that we evaluate the residual threshold for every pair using triple standard deviation. All pixels passing this threshold are labeled as non-trace pixels. Trace Pixel Classification The next step is to determine what trace every trace pixel belongs to. This is done using histograms of hue-saturation space. We perform clustering of h-s blobs and then the back-project result to pixels. To improve the result, we build the histogram using pixels whose bottom-right neighbours have similar color. Since we need the hue difference between pixels, all pixels with similar color should have a similar hue. This is not true for red pixels because they have hue values near both 0 and 180. To tackle this issue, we roll hue values to shift the significant part of the hue histogram to the center. The shift value is found as a minima of the convolution between the hue histogram and the kernel shown below. The idea behind is to “penalize” cases when large histogram values are near 0 or 180. After the histogram is shifted, we find trace pixels whose neighbours have similar hue. This step makes blobs’ edges on the h-s histogram sharper, because pixels near the trace edges are excluded. Similar pixels are found in the next way: Get all trace pixels whose bottom-right pixels are in trace too; Find the 95% percentile of the hue difference; Keep pairs with the hue difference below this level. Then, the mean hue and saturation values are taken for every neighbour pair. The obtained set of the hue and saturation values is used to build ‘filtered’ hue-saturation histogram. This histogram contains less noise produced by the ‘edge’ pixels. After that, the histogram is thresholded by the median value and connected components are found. We build Voronoi diagram and assign its labels to the source histogram. To make the result more robust, we mark the histogram bins below the 10% percentile as unlabeled. Now we get labeled trace pixels like this: There are some mislabeled pixels. To fix them, we perform the following steps: For every possible label we create a binary mask of pixels marked with this label; Gaussian blur is applied to every mask; For every trace pixel we take the index of the mask with the largest value. These steps change labels for the outlier pixels replacing them with the label of surrounding pixels: Chart template detection To determine the chart template we perform the following steps: Recognize all the text presented on the image; Detect axes names, level marks and time marks; Choose the template matching the obtained set of labels. Text Extraction First, we extract all the text char by char, group character images to labels and recognize them. In depth, the next steps are performed: The image is binarized using Otsu method. All trace pixels are made black. So, only the grid and text remain. Connected components are found. Characters are detected using the area and density (the ratio between the area and bounding box area) thresholds and then grouped to text instances. Here is a plot of all text instances found on a typical chart: Text recognition Every text instance is rotated to the horizontal position and passed to Tesseract. First, apply the mask based on the text instance geometry. Then we rotate the image to the horizontal position. To remove the redundant blobs near the text, we draw a line through the middle and use the flood fill to remove the redundant parts. Then we restore the middle row, invert the image and pass it to Tesseract. Template matching Having all text recognized, we find a chart template with the best matching. First, we define a list of all known chart templates. Every template is described by: The number of hours it contains (24 for day, 168 for week, etc.); Text marks appearing on the outer ring; A list of axes; every axis contains: a. Text label; b. Values presented on a chart; c. Possible pen colors. To find the template used on the given image, we check matching of the axis names and range labels in the next way: Group all range labels found on the image by the nearest (by angle) axis name label; Cyclically shift groups to handle combinations with a different starting group. This allows us to handle non-periodical axis sets like ABCABCAB. For every shift: a. Form strings “<axis_name><range_label_1><range_label_2>…” for every group; b. Form a similar string for every axis in the candidate template; c. Calculate string similarity for every pair group-axis; d. Get mean similarity through all pairs; 3. Resulting matching quality is a maximum among all shift values; 4. Take the template with the best quality. After the template is detected, its rotation angle on the image is found by the following steps: Take all labels found in the outer ring of the image; Take the same labels from the detected template; For every found label, expected label and rotation angle, find label-to-label matching quality as a product of string matching ratio and negated angular distance between the expected label and found label, rotated on a given angle. Negation makes the second part of the product increase when the angular distance tends to zero; For every found label and rotation angle, get the best matching expected label; Find a rotation angle with the maximum sum of matching qualities. Axis Matching After template matching we need to find the trace-to-axis correspondence. This is done by color matching. For every trace found a mean color is taken. Then for every possible pair (trace, axis) we find Cartesian distance in RGB space between the mean trace color and the known axis color. Some axes might have two possible pen colors. In this case, the minimal value is taken. Having such distances for all trace-to-axis pairs, we find the best matching by simply iterating all possible correspondences. Pen Trajectory Evaluation So, we have trace points classified by trace. We also know what template was used. This allows us to convert the image coordinates (in pixels) of every point to time and value (in hours and appropriate units: kPa, ℃, etc.). As a result, we get a set of points. If we arrange them in order by time, we have our task solved, right? No… Because now we have tons of points that tell us nothing about the pen’s movement. Let’s look at pixels in time-value plane and a corresponding chart fragment: It is easy to tell how the pen moved on the left and right ends of this excerpt, because we clearly see the line. But what about the middle part? How did the pen travel here? Intuitively, we can say that the pen moved up and down, shifting right a little every time. But how to restore this movement? To solve this task, the next steps are performed. Shear compensation First, we remove inner-points. Only border points are kept. This dramatically decreases the number of points for further processing. After that, we compensate the trace shear. This shear happens when the pen is placed incorrectly or is not fixed in the plotting head. To tackle it, we iterate over possible shear coefficients. For each one, we find an area of min-max envelope. The coefficient with the smallest result is chosen and applied to trace points. The image below shows the bounding area for shearing coefficients -0.01 and 0.01. Here is the result of shearing with the optimal coefficient: Improving envelope Next, we improve the obtained min-max envelope. Initially, all bins are uniformly distributed. Sometimes thin point “columns” appear on the edge between the bins, producing two bins instead of one. Simple movement of all bins won’t help us, because having one “column” aligned, we might get another one on an edge. Below is an example of two peaks and the envelope with step 2. To solve this issue, we improve the position of the bins by means of the next steps: Generate 10 envelopes for different bin offsets; Get max-to-min differences for each one; Apply three-item sliding mean; Having 10xN matrix and treating it as cost, we use forward-backward algorithm to obtain the sequence of indices (“path”) that minimizes the total area of the envelope. After these steps, we get the envelope with a smaller area than the initial one: Outlier removal Next, we remove the outliers. Sometimes grid pixels, hand marks or other things are mislabeled as trace points. For example, here we have added three outlier blobs: Applying the min-max envelope gives us an inadequate result: To fix this, we split high blocks into vertical “chunks” with a limited height. Now for every bin more than one chunk can exist. For example, here are three such cases: After that we find optimal indices of chunks for every vertical block. We use forward-backward algorithm with a vertical distance between the neighbouring chunks as cost function. This gives us a sequence of chunks covering inlier points. Next, we get an optimal envelope for inliers: Pen trajectory evaluation Having done this, we are ready to evaluate the pen trajectory. We assume the linear pen motion inside every bin, changing the movement direction between the bins. The goal is to maximize the area of the bin covered by the pen motion. For example, here is a possible envelope, pen path and coverage ratio for every bin: To maximize the coverage area, we use forward-backward algorithm with ten points per bin edge as possible states and one minus bin coverage as the cost of step. Here is the result applied to the example above: And to the real-world example: The red line on the image above shows the evaluated pen trajectory. It follows the thin line where needed and covers dense point areas using the up-down movement. Conclusion The algorithm described above successfully handles a wide range of input data from thin lines to dense “hatching”. The produced data allows us to calculate the gas flow characteristics without any human interaction, increasing the robustness of the whole chart handling process. By Ilya Sivkov

ChartRecorder: Computer Vision for Gas Flow Charts

Introduction: Flow of the program depends on the value of the various variables we use in our code. Depending on the values of the variables in our program, we navigate the flow of execution as we needed / required. For example if we are using any collection types, it is hard to perform logic when the collection is modified or changed. i.e when new items are added, deleted, or modified. We can still manage this kind of scenarios in many ways. One of them is using Notification-Centre, To notify us whenever the change occurs in property value. But if want to check on many properties in our code, this (using Notification Centre) way lead us to heavy chunk of code, to call for all those properties we need to check on. The Other way which is better In this kind of scenarios is (also Apple using this in its libraries a lot) known as (Key Value Observing), which is also directly related to another powerful mechanism called (Key Value Coding). KVO KVC Note: Any property we want to observe for changes must be KeyValueCoding (KVC)complaint. Both of these KVO & KVC, provide an efficient way to write our code. Now lets get started with KVC and then KVO. KVC: is a form of coding that allows you to access an object’s properties indirectly, using strings to access them instead of the property’s accessors or instead of accessing the variables directly. To enable such mechanism, your classes must comply to the NSKeyValueCoding informal protocol. (OR) KVC Key-value coding is a mechanism for accessing an object’s properties indirectly, using strings to identify properties, rather than through invocation of an accessor method or accessing them directly through instance variables. Ex: class Profile: NSObject { var firstName: String var lastName: String var customProfile: Profile } If we want assign the values for the above declared variables, in the class init() or somewhere in the class file, normally we do like the following: self.firstName = “Robert” self.lastName = “Stark” With KVC; we do like the following: self.setValue: “ Robert ” for key: “ firstName ” //self.setValue: Any for key: key/KeyPath self.setValue: “ Stark ” for key: “ lastName ” To retrieve the values of KVC properties we use like the following: let robertLastName = self.value( forKey: “ lastName ” ) The above KVC working way is like working with Dictionaries in swift. Right? Here, In KVC; Instead of directly , or using (#If available) , we are doing in a way simply values to assigning the values to properties setter methods of objects assigning keys/keyPaths. So we use keys and values , this technique is called Key Value Coding(KVC). every class that is defined in the framework and that inherits from conforms to the protocol. Note: There is protocol known as NSKeyValueCoding informal protocol , which is compulsory to work with KVC . Our classes must be confirmed to this protocol, in order to use KVC & KVO . NSObject confirms to this protocol. So Foundation NSObject NSKeyValueCoding What is a key & KeyPath ? Key : simply “ Key ” specifies a single property, the one we want to set a value or get one from . So its name should be the same as the property’s name. Ex: self.setValue: “ Stark ” for key: “ lastName ” KeyPath : A KeyPath is formed with the dot-syntax by following the substrings, so it is not a single word/string. Key-path represents all the properties of an object, which comes in the way until to reach the desired value/property . Ex: var myProfile: Profile self.setValue: “ Baratheon ” for key: “ myProfile. customProfile.lastName ” KVC : Here, we are going to create a single view application, to work with the KVC & KVO. This project has two example projects, but the one we are exploring here is not related with any UI. The other one is updates a UILable on ViewController’s view when the text changes. The Project link will be shared at the bottom, so you can explore easily yourself. First, create a single view application in swift To use KVC our classes need to confirm to NSKeyValueCoding protocol. So by confirming to NSObject, we can achieve this step. UIViewController is already confirms to this NSObject, so We can call the methods from this protocol without any setup. In Xcode, In your project create a new swift file, with the name Children with NSObject as a base class(Children inherits from NSObject). Then In that class file, declare two properties named ‘ and ‘ by placing ‘ ’ keywords at the front of the declaration_._ So why do we need to do this??? name’ age’ @objc dynamic → In swift, there are many keywords/ attributes to help the compiling, runtime specifications, access controlling etc. Ex: @escaping, @available, etc. Likewise, It also defines a range of declaration modifiers, to modify the declarations of properties/class-members. For example, by marking a class declaration with the‘ keyword, we inform the compiler that the class cannot be subclassed. This allows the compiler to make a number of optimisations to increase performance. ‘ is also a declaration modifier we use in swift. final’ dynamic’ , is one of the cool features in Objective-C. It simply means that the Objective-C runtime decides at runtime which implementation of a particular method or function it needs to invoke. For example, if a subclass overrides a method of its superclass, dynamic dispatch figures out which implementation of the method needs to be invoked, that of the subclass or that of the parent class. This is a very powerful concept. Dynamic dispatch Swift uses the Swift runtime whenever possible. The result is that it can make a number of optimisations. While Objective-C solely relies on dynamic dispatch, Swift only opts for dynamic dispatch if it has no other choice. If the compiler can figure out at compile time which implementation of a method it needs to choose, it wins a few nanoseconds by opting out of dynamic dispatch. Swift runtime chooses other options, such as and , over whenever possible. It does this to increase performance.Static and virtual dispatch are much faster than dynamic dispatch. Even though we are talking nanoseconds, the net result can be dramatic. Many features we have come accustomed to are only possible because of the dynamic Objective-C runtime, including Core Data and Key-Value Observing. static virtual dispatch dynamic dispatch Dynamic Dispatch By applying the ‘ declaration modifier to a member of a class, you tell the compiler that dynamic dispatch should be used to access that member. dynamic’ By prefixing a declaration with the ‘ keyword, the declaration is implicitly marked with the attribute. The attribute makes the declaration available in Objective-C, which is a requirement for it to be dispatched by the Objective-C runtime. dynamic’ objc objc ‘ declaration modifier can only be used for members of a class. Structures and enumerations don't support inheritance, which means the runtime doesn't have to figure out which implementation it needs to use. dynamic’ So to use KVC & KVO in swift, for the properties we want to observe in KVO we need to declare them with @objc dynamic keyword. Now, we have two properties in our class, Initialise/define them in initialiser method. Children Now in ViewController class file, declare three Children Instances like the following: First we use of ‘ object. In the viewDidLoad() method, we initialise the child1 object, and then assign the values to its properties. child1’ If you print the‘ objects properties like name and age you will get the assigned values in result. child1’ Now, we will use the KVC methods to do the same thing. In the above snippet, In the first couple of rows we set the desired values to both properties using the method. observe the key strings are the same to the properties’s names. setValue:forKey: Next, we perform the exact opposite task. We extract the values out of the properties using the method, and we assign them to two local variables. And then we printed the values in the console. The result is the same as the previous one. valueForKey: Note: If we give keys different to the property names, The app will crash. When writing KVC-compliant code it’s really important to take care so the key strings actually match to the property names, otherwise the app will simply fall apart. This is not a case when dealing directly with properties, as there’s no chance to do any mistake on their names; the compiler would throw an error in such a case which would drive us to fix it. With all the above, we have managed to see how to write KVC-styled code, and how to set and get values using keys. In the next part we will complete the project. working with key-paths: Now go to the Children class, and add the following property. @objc dynamic var child: Children? in the method. Now, add the next lines that initialise the related objects and assign their initial values: viewDidLoad In the above snippet, in the first couple of rows we just initialised the‘ _child2 ’_object and its ‘ property. Next we set the values to the properties and of . For the property of , we used key-path to set the values. Observe closely. we can retrieve the values to check if the assignment was successful: Uncomment the print statements for that. child’ name age child2 child child2 Now we will see what if the child of the child has a child too… If you want to check the values/result use the print statements for the above snippet. So far, we learned how to write KVC complaint code using keys and key-paths. Next we will learn Observing the Property’s value changes: KVO : Here we will see what actions should be taken in order to be able to track down changes on properties. First of all, let me introduce you as a list the steps needed to implement KVO: 1. The class of which you want to observe its properties must be KVO compliant. That means: The class must be KVC compliant according to what we have already seen in the introduction and in the previous section. The class must be able to send notifications either automatically, or manually (we will see more about that later). 2. The class that will be used to observe the property of another class should be set as . observer 3. A special method named should be implemented to the observing class. observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) Let’s see everything one by one. The most important thing when we want to observe for changes of a property, is to our class observe for these changes. This is done more or less with as with the casual notifications ( ), but using another method. This method is the make NSNotifications addObserver(<#T##observer: NSObject##NSObject#>, forKeyPath: <#T##String#>, options: <#T##NSKeyValueObservingOptions#>, context: <#T##UnsafeMutableRawPointer?#>) Here we observe the value changes for and properties of a object. So in viewWillAppear() method, add the observers for the child1 object. name age child1 The parameters the above method accepts are: : This is the observing class, usually the object. addObserver self : I guess you can understand what’s this for. It is the string you used as a key or a key path and matches to the property you want to observe. Note that you specify here either a single key, or a key path. forKeyPath : an array of values. options NSKeyValueObservingOptions : This is a pointer that can be used as a unique identifier for the change of the property we observe. Usually this is set to or . We’ll see more about this later. context nil NULL Now that we have made our class able to observe for any changes in the above two properties, we must implement the method. Its implementation is mandatory, and it has one great disadvantage. That is the fact that is called for every KVO change, and if you observe many properties then you must write a lot of statements so as to take the proper actions for each property. However, this can be easily overlooked as the benefits of the KVO are greater than this limitation. observeValueForKeyPath:ofObject:change:context: if The above method will be called every time change in the value of properties which are added to the observer. Here by key-path parameter, we printed the new and old values of the name and age properties. To test this functionality, wire-up an action for a UIButton to your viewController, and in that action method add the following lines of code: Super! After we have set a new value to the and properties of the object, we received the notification, and the messages we asked for to be displayed were shown on the debugger. As you see, both the previous and the new value are included in the dictionary. name age child1 From the dictionary you can extract any value you want (if needed), but the most important of all is that it’s super-easy to be notified about changes in properties. change add the following lines of code in the button action method and run the program. If you observe the result, we receive two notifications regarding changes to the property. But that seems confusing, because even though we know the object that each notification belongs to, programmatically we can do nothing to determine the object that sent the notification. So, how do we face that, and how can we programmatically be a 100% sure about the object that the changed property belongs to? age The answer to the above question is one: We will make use of the argument of the method. I have already mentioned before that the purpose of the context is to uniquely identify a change on a property, so it’s the best tool we have at our disposal. context addObserver(<#T##observer: NSObject##NSObject#>, forKeyPath: <#T##String#>, options: <#T##NSKeyValueObservingOptions#>, context: <#T##UnsafeMutableRawPointer?#>) Here I defined two contexts for each child1 and child2: Note that the context value for each observed property must be a global variable, because it has to be accessible from both the and the methods. addObserver… observeValueForKeyPath… Now modify the code in child1 and child2 add observer methods, by passing the context parameter. And then we need to modify the observe method as well like the following: This way we can identify the changed property with help of passed context parameter. So we managed to programmatically specify each changed property. Lastly and before we reach at the end of this chapter, it’s also quite important at some point to remove the observers you add. There is not a recipe on where you should do that. For instance, in many cases it would be useful to do that in the , after you have handled a received notification. In other cases, you should do so upon the dismissal of a view controller. Generally, it’s up to your application’s structure the decision you will make about that. In this example, we will do it in the method. Here it is: For observeValueForKeyPath:ofObject:change:context: viewWillDissapear: child2, I removed observer in the above snippet. Automatic and Manual Notifications By default, the system sends a notification every time a property gets changed when you observe using KVO. This is suitable in most cases, however there are times that we don’t want to get a notification once a change has happened, but after a bunch of changes have taken place in multiple properties or at a later time. Thankfully, iOS SDK provides us with some quite convenient methods that gives us control over the notifications, so we can manually send them whenever it’s actually needed. Before we get into more details, let me just say that using the method you’ll see right next is not mandatory. On the contrary, you may implement it if and when it is really necessary. Getting into the point now, in order to control the notifications that are sent upon property changes, you must implement the _automaticallyNotifiesObserverForKey:_class method. The parameter it accepts is a string representation of the key of the property for which you need to control the notification, and it returns a boolean value. In case that you don’t want a notification to be sent after the observed property’s value has been changed, then the method must return false. In any other case, you should let iOS decide about the notifications. In practice, let’s suppose that we don’t want a notification to be posted when the property of the class get changed. With that in mind, here’s the implementation of that method in the class name Children Children In the clause, we call the same method using the class in order to let iOS handle all the keys that we haven’t explicitly added here, and the value that we get back is the one that we return at the end. else super If you run the app at this point, you’ll find out that no message regarding the name changing is appeared on the debugger. Of course, this is what we desire, so we’ve managed to achieve our goal. But, have we really done it? Well, as you understand by returning false in the above method for the specific key, we’ve only managed to stop the relevant notifications from being sent. And of course, this doesn’t mean manual notifications, it means no notifications at all! In order to send a notification when we decide so, we must use two other methods. These are the and the . When using them, the must be called first, then the new value must be assigned to the property, while the should be called at the end. To test the manual actions, add these lines in the action method of viewcontroller. willChangeValueForKey: didChangeValueForKey: willChangeValueForKey: didChangeValueForKey: If you run the app now, the message regarding the name change is appeared on the debugger, and that means that we’ve sent the notification manually with success! Actually, the notification is sent after the method is invoked, therefore place that method wherever you feel appropriate for receiving notification. didChangeValueForKey: That’s what exactly we have been expecting from the app! As you can see, we’ve managed to control the notification sending point, and all that with a little effort! Note that between the and the methods, you can have more than one property values assigned. willChangeValueForKey: didChangeValueForKey: The and the methods are not mandatory to be used as shown above. They can be implemented in the class (or in your own class) as well. willChangeValueForKey: didChangeValueForKey: Children Please click for the project describing above tutorial. here References: http://michael-brown.net/2017/swift-and-kvo-context-variables/ https://www.appcoda.com/understanding-key-value-observing-coding/ https://cocoacasts.com/key-value-observing-kvo-and-swift-3 https://www.ralfebert.de/ios-examples/swift/property-key-value-observer/ https://blog.scottlogic.com/2015/02/11/swift-kvo-alternatives.html If you like my tutorials please follow me on . You can contact me through or accounts. medium my-twitter linkedIn Thanks for reading. Will get better soon with more tutorials. See you!!!

KVO & KVC In swift

KVO y KVC en rápido

These are the words that I had to mutter quite a few times in my career, at the dissatisfaction of how pagination had been implemented on several projects. Still, that dissatisfaction is nothing compared to how I feel when I occasionally had to ask “why is this API not paginated..?” So, taking a break from my usual Serverless ramblings, let’s talk about pagination :-) Unidirectional and Bidirectional Pagination Generally speaking, I see two common types of paginations: simple, unidirectional, paging through a static set of results that are too long or inefficient to return in one go — e.g. list of twitter followers, or list of Google search results bidirectional paging through a feed or stream of some sorts where new results can be added after you received the first page of results — e.g. your twitter timeline, or notifications Avoid leaky abstraction A common mistake I see is that the paginated API requires the caller to provide the “key” it uses to sort through the results, which creates a leaky abstraction. The caller must then understand the underlying mechanism the service uses to page its results — e.g. by timestamp, or alphabetical order. DynamoDB’s API is a good example of this. To page through the query results, the caller must specify the in subsequent requests. However, the service does return the in the response too. Query ExclusiveStartKey LastEvaluatedKey So, in practice, you can treat the as a token, or cursor, which you simply pass on in the next request. Except, it’s not just a token, it’s an actual sort key in the table, and the attribute names already gave the implementation details away anyway. almost LastEvaluatedKey DynamoDB On its own, this is not a big deal. However, it often has the unfortunately knock-on effect of encouraging application developers to build their application level paginations on top of this implementation detail. Except this time around, they’re not returning the in the response and the client is now responsible for tracking that piece of information. LastEvaluatedKey Congratulations, the underlying mechanic your database uses to support pagination has now leaked all the way to your front end! Make paging intent explicit and consistent Another common trend I see is that you have to send the same request parameters to the paginated API over and over, for example: max no. of results per page the direction of pagination (if bidirectional) the original query (in ’s case, this includes a no. of attributes such as , , and ) DynamoDB FilterExpression KeyConditionExpression ProjectionExpression IndexName I won’t call this one a mistake as it is by design, but more often than not it strikes me as a instead. sometimes consequence of lack of design In all the paginated APIs I have encountered, the intended behaviour is always to fetch the next for a query, not to start a different query midway through. That just wouldn’t make sense, and you probably can’t even call that pagination, more like navigation! I mean, set of results when was the last time you start a DynamoDB query, and then have to change any of the request parameters midway through paginating through the results? That said, there are legitimate reasons for changing the of pagination from a previously received page. More on this when we discuss bidirectional paging further down the article. direction Unidirectional paging with cursor For unidirectional pagination, my preferred approach is to use a simple cursor. The important detail here is to . make the **cursor** meaningless As far as the client is concerned, it’s just a blob the server returns in the response . The client shouldn’t be able to derive any implementation details from it, and the only thing it can to do with this is to send it along in the next request. when there are more results to fetch afford cursor fig. 1 — flow of request & responses for a series of paginated requests But how does the API know where to start fetching the next page from? A simple way to do this is: create a JSON object to capture the data needed to fetch next page — e.g. if you’re using DynamoDB, then this can be the request object for the next page (including the ) ExclusiveStartKey base64 encode the JSON string return the base64 blob as cursor When we receive the request to fetch the next page, we can apply the reverse process to get back the request object we created earlier. Isn’t that gonna leak even more information — e.g. that you’re using DynamoDB, the table name, the schema, etc. — if someone just base64 decode your blob? Absolutely, which is why you might also choose to the JSON first. You also don’t have to use the DynamoDB request as base. encrypt query Notice in both and the client only sends the in the subsequent requests? fig. 1 fig. 2 cursor . This is by design As I mentioned earlier, the client has already told us the query in the first request, the pagination mechanism should provide a way for fetching subsequent pages of the results. only Which, to me, means it should not any other behaviour (there is that word again, to see how the idea of applies to API design) and therefore do not require any other information besides the from the previous response. afford read here affordance cursor This in turn, means we need to capture the original query, or intent, in the so we can construct the corresponding DynamoDB request. Or, we could just capture the actual DynamoDB request in the which seems like a simple, practical solution here. cursor cursor fig. 2 — interaction between client, API and DynamoDB Bidirectional paging with cursor(s) With bidirectional paging, you need to be able to page forward in time (when new tweets are added to your timeline) as well as backward (to fetch older tweets). So a simple string would no longer suffice, instead we need 2 cursors, one for each direction, for example… cursor {"before": "ThlNjc5MjUwNDMzMA...","after": "ADfaU5ODFmMWRiYQ..."} Additionally, when paging forward, even when there are no more results right now we still have to return a cursor as new results can be added to the feed later. So we should also include a pair of boolean flags in the response. {"before": "ThlNjc5MjUwNDMzMA...","hasBefore": true,"after": "ADfaU5ODFmMWRiYQ...","hasAfter": true} When the client pages forward in time and receives as then it knows there are no more results available right now. It can therefore stop fetch the next page of results, and be more passive and only poll for new results periodically. hasAfter false actively Let’s run through a simple example, imagine if you’re fetching the tweets in your timeline where the API would return the latest tweets first. fig. 3 — paginate backward in time to fetch older data the client makes a first request API responds with a object, is because the API has responded with the latest results, but is as there are older results available cursor hasAfter false hasBefore true the client makes a second request and passes the cursor in the request, making its intention clear and unambiguous only before API responds with another object where this time both and are given that we’re right in the middle of this stream of results cursor hasBefore hasAfter true the client makes a third and last request, again passing only the cursor received from the previous response before API responds with a object where is because we have now received the oldest result available cursor hasBefore false Ok, now let’s run through another example, this time we’ll page forward in time instead. fig. 4 — paginate forward in time to fetch newer data the client makes a first request API responds with a object, is because the API has responded with the latest results, but is as there are older results available cursor hasAfter false hasBefore true some time has passed, and more results have become available the client makes a second request, and passes the cursor in the request, making its intention clear and unambiguous only after API responds with the newer results that the client has not received already, and is as these are the latest results available at this moment in time; should the client page backward (in time) from this response then it’ll receive the same results as the first response from the API only cursor.hasAfter false Now, let’s circle back to what I mentioned earlier regarding the occasional need to change direction midway through pagination. The reason we need pagination is because it’s often impractical, inefficient and in some cases impossible to return all available results for a query — e.g. at the time of writing Katy Perry has 108M Twitter followers, trying to retrieve all her followers in one request-response cycle would have crashed both the server and the client app. Besides limiting how much data can be returned in one request-response cycle, we also need to . place a upper bound on how much data the client app would cache in order to protect the user experience and prevent the client app from crashing That means, at some point, as the user keeps scrolling through older tweets, the client would need to start dropping data it has already fetched or risk running out of memory. Which means, when the user scrolls back up to see the latest tweets, the client would need to re-fetch pages that had been dropped and hence reversing the original direction of the pagination. Fortunately, the scheme outlined above is flexible enough and allows you to do just that. Every page of results has an associated that allows you to fetch the next page in either direction. So, in the case where you need to re-fetch a dropped page, it’s as simple as making a paginated request with the cursor of the latest page you have cached. cursor after Dealing with “gaps” Staying with the Twitter example. If you open up the Twitter mobile app after some time, you’ll see the tweets that has already been cached but the app also recognizes that a lot of time has passed that it’s not feasible to paginate from the cached data all the way to the latest. Instead, the client would fetch the latest tweets with a non-paginated request. As you scroll down, the client automatically fetch older pages as per and gradually fill in the gap until it joins up with the cached data. can fig. 3 The behaviour of the Twitter mobile app has changed over time, and another tactic I have seen is to place a visual (clickable) marker for the missing tweets in the timeline. This makes it an explicit action by the user to start paging through older tweets to fill in the gap. So there you have it, a simple and effective way to implement both unidirectional and bidirectional paginated APIs, hope you have found it useful! Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here

Stories, News & Mentions about 1FLOW

1flow’s Stories on HackerNoon